2024-03-26T01:43:46.294 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-26T01:43:46.295 DEBUG:teuthology.run:Teuthology command: teuthology --description orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621931 --name teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi --verbose -- /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621931/orig.config.yaml 2024-03-26T01:43:46.332 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T01:43:46.419 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621931 branch: squid description: orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} email: ceph-qa@ceph.com first_in_suite: false job_id: '7621931' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: debug ms: 1 global: mon election default strategy: 3 ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on pool no app: false osd: debug ms: 1 debug osd: 20 osd class default list: '*' osd class load list: '*' osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - reached quota - but it is still running - overall HEALTH_ - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: cephadm-package install: ceph: flavor: default sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a extra_packages: - cephadm selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: 9f04c91b77e4619963d41730d8cf756e3a6152de owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 1822 sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a sleep_before_teardown: 0 subset: 53/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 9f04c91b77e4619963d41730d8cf756e3a6152de targets: smithi072.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ5gnemEUL2Ea2CURc2ZLotym3ZebY8KEH1+QZ84LeAnn4Sarfk9WPTqPEhmxdxR9RWS89EgCRi48t56sFPdlb4= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJR4IJEppupGoPqwaxFcNA3mN75efbtlVIdC+QPBtGM33IkD91QvLX0vXCubmrVNl0+RXcjRCikGpXKl85e5xuM= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - workunit: clients: client.0: - rados/test.sh - rados/test_pool_quota.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_21:08:11 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-26T01:43:46.419 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa; will attempt to use it 2024-03-26T01:43:46.420 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa/tasks 2024-03-26T01:43:46.420 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-26T01:43:46.421 INFO:teuthology.task.internal:Checking packages... 2024-03-26T01:43:46.445 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5e70124b46c2a671efcaae4690cd9ea72670710a' 2024-03-26T01:43:46.446 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-26T01:43:46.446 INFO:teuthology.packaging:ref: None 2024-03-26T01:43:46.446 INFO:teuthology.packaging:tag: None 2024-03-26T01:43:46.446 INFO:teuthology.packaging:branch: squid 2024-03-26T01:43:46.446 INFO:teuthology.packaging:sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T01:43:46.446 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-26T01:43:46.595 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1863.g9f04c91b 2024-03-26T01:43:46.597 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-26T01:43:46.603 INFO:teuthology.task.internal:no buildpackages task found 2024-03-26T01:43:46.604 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-26T01:43:46.610 INFO:teuthology.task.internal:Saving configuration 2024-03-26T01:43:46.622 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-26T01:43:46.628 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T01:43:46.653 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi072.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621931', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 01:38:27.513298', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ5gnemEUL2Ea2CURc2ZLotym3ZebY8KEH1+QZ84LeAnn4Sarfk9WPTqPEhmxdxR9RWS89EgCRi48t56sFPdlb4='} 2024-03-26T01:43:46.677 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621931', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 01:38:27.511347', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJR4IJEppupGoPqwaxFcNA3mN75efbtlVIdC+QPBtGM33IkD91QvLX0vXCubmrVNl0+RXcjRCikGpXKl85e5xuM='} 2024-03-26T01:43:46.677 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-26T01:43:46.683 INFO:teuthology.task.internal:roles: ubuntu@smithi072.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-26T01:43:46.684 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-26T01:43:46.684 INFO:teuthology.run_tasks:Running task console_log... 2024-03-26T01:43:46.777 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f2c1dd710d0>, signals=[15]) 2024-03-26T01:43:46.777 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-26T01:43:46.784 INFO:teuthology.task.internal:Opening connections... 2024-03-26T01:43:46.784 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi072.front.sepia.ceph.com 2024-03-26T01:43:46.787 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:43:46.877 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2024-03-26T01:43:46.878 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:43:46.966 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-26T01:43:46.973 DEBUG:teuthology.orchestra.run.smithi072:> uname -m 2024-03-26T01:43:46.991 INFO:teuthology.orchestra.run.smithi072.stdout:x86_64 2024-03-26T01:43:46.991 DEBUG:teuthology.orchestra.run.smithi072:> cat /etc/os-release 2024-03-26T01:43:47.047 INFO:teuthology.orchestra.run.smithi072.stdout:NAME="CentOS Stream" 2024-03-26T01:43:47.047 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION="9" 2024-03-26T01:43:47.047 INFO:teuthology.orchestra.run.smithi072.stdout:ID="centos" 2024-03-26T01:43:47.047 INFO:teuthology.orchestra.run.smithi072.stdout:ID_LIKE="rhel fedora" 2024-03-26T01:43:47.047 INFO:teuthology.orchestra.run.smithi072.stdout:VERSION_ID="9" 2024-03-26T01:43:47.048 INFO:teuthology.orchestra.run.smithi072.stdout:PLATFORM_ID="platform:el9" 2024-03-26T01:43:47.048 INFO:teuthology.orchestra.run.smithi072.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T01:43:47.048 INFO:teuthology.orchestra.run.smithi072.stdout:ANSI_COLOR="0;31" 2024-03-26T01:43:47.048 INFO:teuthology.orchestra.run.smithi072.stdout:LOGO="fedora-logo-icon" 2024-03-26T01:43:47.048 INFO:teuthology.orchestra.run.smithi072.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T01:43:47.048 INFO:teuthology.orchestra.run.smithi072.stdout:HOME_URL="https://centos.org/" 2024-03-26T01:43:47.048 INFO:teuthology.orchestra.run.smithi072.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T01:43:47.048 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T01:43:47.048 INFO:teuthology.orchestra.run.smithi072.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T01:43:47.049 INFO:teuthology.lock.ops:Updating smithi072.front.sepia.ceph.com on lock server 2024-03-26T01:43:47.071 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2024-03-26T01:43:47.088 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2024-03-26T01:43:47.088 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2024-03-26T01:43:47.145 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="CentOS Stream" 2024-03-26T01:43:47.145 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="9" 2024-03-26T01:43:47.145 INFO:teuthology.orchestra.run.smithi173.stdout:ID="centos" 2024-03-26T01:43:47.145 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="rhel fedora" 2024-03-26T01:43:47.145 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="9" 2024-03-26T01:43:47.146 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el9" 2024-03-26T01:43:47.146 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T01:43:47.146 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31" 2024-03-26T01:43:47.146 INFO:teuthology.orchestra.run.smithi173.stdout:LOGO="fedora-logo-icon" 2024-03-26T01:43:47.146 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T01:43:47.146 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://centos.org/" 2024-03-26T01:43:47.146 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T01:43:47.146 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T01:43:47.146 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T01:43:47.146 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2024-03-26T01:43:47.171 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-26T01:43:47.180 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-26T01:43:47.186 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-26T01:43:47.187 DEBUG:teuthology.orchestra.run.smithi072:> test '!' -e /home/ubuntu/cephtest 2024-03-26T01:43:47.189 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2024-03-26T01:43:47.204 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-26T01:43:47.210 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-26T01:43:47.210 DEBUG:teuthology.orchestra.run.smithi072:> test -z $(ls -A /var/lib/ceph) 2024-03-26T01:43:47.245 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2024-03-26T01:43:47.336 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-26T01:43:47.394 INFO:teuthology.run_tasks:Running task kernel... 2024-03-26T01:43:47.408 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-26T01:43:47.408 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-26T01:43:47.408 DEBUG:teuthology.orchestra.run.smithi072:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T01:43:47.409 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T01:43:47.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T01:43:47.426 DEBUG:teuthology.orchestra.run.smithi072:> uname -r 2024-03-26T01:43:47.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T01:43:47.427 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-03-26T01:43:47.482 INFO:teuthology.orchestra.run.smithi072.stdout:5.14.0-361.el9.x86_64 2024-03-26T01:43:47.483 INFO:teuthology.task.kernel:Running kernel on smithi072: 5.14.0-361.el9.x86_64 2024-03-26T01:43:47.483 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum install -y kernel 2024-03-26T01:43:47.484 INFO:teuthology.orchestra.run.smithi173.stdout:5.14.0-361.el9.x86_64 2024-03-26T01:43:47.484 INFO:teuthology.task.kernel:Running kernel on smithi173: 5.14.0-361.el9.x86_64 2024-03-26T01:43:47.484 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum install -y kernel 2024-03-26T01:43:49.696 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-26T01:43:50.580 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-26T01:43:51.163 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - BaseOS 5.9 MB/s | 2.0 MB 00:00 2024-03-26T01:43:51.981 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-26T01:43:52.220 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-26T01:43:54.099 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - BaseOS 4.6 MB/s | 8.0 MB 00:01 2024-03-26T01:44:01.920 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 2.2 MB/s | 19 MB 00:08 2024-03-26T01:44:05.568 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - AppStream 1.8 MB/s | 19 MB 00:10 2024-03-26T01:44:10.977 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - CRB 3.2 MB/s | 6.1 MB 00:01 2024-03-26T01:44:12.422 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 16 kB 00:00 2024-03-26T01:44:13.020 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 836 kB/s | 6.1 MB 00:07 2024-03-26T01:44:14.545 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-03-26T01:44:15.736 INFO:teuthology.orchestra.run.smithi072.stdout:Extra Packages for Enterprise Linux 6.5 MB/s | 21 MB 00:03 2024-03-26T01:44:17.355 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 7.7 MB/s | 21 MB 00:02 2024-03-26T01:44:20.559 INFO:teuthology.orchestra.run.smithi072.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-03-26T01:44:22.346 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-26T01:44:22.716 INFO:teuthology.orchestra.run.smithi072.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T01:44:22.717 INFO:teuthology.orchestra.run.smithi072.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T01:44:22.822 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:44:22.838 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:44:22.838 INFO:teuthology.orchestra.run.smithi072.stdout: Package Architecture Version Repository Size 2024-03-26T01:44:22.838 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:44:22.838 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-26T01:44:22.838 INFO:teuthology.orchestra.run.smithi072.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T01:44:22.838 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-03-26T01:44:22.838 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T01:44:22.838 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T01:44:22.838 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T01:44:22.838 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:44:22.838 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-26T01:44:22.839 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:44:22.839 INFO:teuthology.orchestra.run.smithi072.stdout:Install 4 Packages 2024-03-26T01:44:22.839 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:44:22.839 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 76 M 2024-03-26T01:44:22.839 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 123 M 2024-03-26T01:44:22.839 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-26T01:44:23.245 INFO:teuthology.orchestra.run.smithi072.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 479 kB/s | 66 kB 00:00 2024-03-26T01:44:24.470 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T01:44:24.470 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T01:44:24.561 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:44:24.570 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:44:24.570 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-03-26T01:44:24.570 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:44:24.570 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-26T01:44:24.570 INFO:teuthology.orchestra.run.smithi173.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T01:44:24.570 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-26T01:44:24.571 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T01:44:24.571 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T01:44:24.571 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T01:44:24.571 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:44:24.571 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-26T01:44:24.571 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:44:24.571 INFO:teuthology.orchestra.run.smithi173.stdout:Install 4 Packages 2024-03-26T01:44:24.571 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:44:24.572 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 76 M 2024-03-26T01:44:24.572 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 123 M 2024-03-26T01:44:24.572 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-26T01:44:24.756 INFO:teuthology.orchestra.run.smithi173.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 936 kB/s | 66 kB 00:00 2024-03-26T01:44:25.265 INFO:teuthology.orchestra.run.smithi173.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 26 MB/s | 15 MB 00:00 2024-03-26T01:44:25.757 INFO:teuthology.orchestra.run.smithi173.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 31 MB/s | 33 MB 00:01 2024-03-26T01:44:26.207 INFO:teuthology.orchestra.run.smithi173.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 19 MB/s | 27 MB 00:01 2024-03-26T01:44:26.207 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:44:26.207 INFO:teuthology.orchestra.run.smithi173.stdout:Total 46 MB/s | 76 MB 00:01 2024-03-26T01:44:26.522 INFO:teuthology.orchestra.run.smithi072.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 4.4 MB/s | 15 MB 00:03 2024-03-26T01:44:26.872 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-26T01:44:27.256 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-26T01:44:27.257 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-26T01:44:27.339 INFO:teuthology.orchestra.run.smithi072.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 6.7 MB/s | 27 MB 00:04 2024-03-26T01:44:28.267 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-26T01:44:28.268 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-26T01:44:28.323 INFO:teuthology.orchestra.run.smithi072.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 6.4 MB/s | 33 MB 00:05 2024-03-26T01:44:28.323 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:44:28.323 INFO:teuthology.orchestra.run.smithi072.stdout:Total 14 MB/s | 76 MB 00:05 2024-03-26T01:44:28.991 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-26T01:44:29.373 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-26T01:44:29.373 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-26T01:44:30.362 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-26T01:44:30.362 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-26T01:44:30.719 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-26T01:44:31.321 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T01:44:31.407 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:44:32.795 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-26T01:44:33.467 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T01:44:33.561 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:44:34.158 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:44:34.285 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:44:36.288 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:44:36.480 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:44:39.010 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:44:39.051 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:44:41.153 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:44:41.202 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:44:43.790 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:44:45.878 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:45:13.009 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:45:13.010 INFO:teuthology.orchestra.run.smithi173.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T01:45:13.010 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:45:13.054 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:45:13.435 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:45:13.435 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T01:45:13.435 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:45:13.435 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:45:13.843 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:45:13.844 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:45:13.844 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-26T01:45:13.844 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T01:45:13.844 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T01:45:13.844 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T01:45:13.844 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T01:45:13.844 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:45:13.844 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:45:14.178 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T01:45:14.283 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T01:45:14.284 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T01:45:14.284 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T01:45:14.284 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T01:45:14.284 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T01:45:14.284 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-26T01:45:14.284 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi173.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T01:45:14.285 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T01:45:14.364 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T01:45:14.365 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T01:45:14.365 DEBUG:teuthology.orchestra.run.smithi173:> sudo rpm -qi grub2-tools 2024-03-26T01:45:14.402 INFO:teuthology.orchestra.run.smithi173.stdout:Name : grub2-tools 2024-03-26T01:45:14.402 INFO:teuthology.orchestra.run.smithi173.stdout:Epoch : 1 2024-03-26T01:45:14.402 INFO:teuthology.orchestra.run.smithi173.stdout:Version : 2.06 2024-03-26T01:45:14.402 INFO:teuthology.orchestra.run.smithi173.stdout:Release : 68.el9 2024-03-26T01:45:14.402 INFO:teuthology.orchestra.run.smithi173.stdout:Architecture: x86_64 2024-03-26T01:45:14.402 INFO:teuthology.orchestra.run.smithi173.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T01:45:14.402 INFO:teuthology.orchestra.run.smithi173.stdout:Group : Unspecified 2024-03-26T01:45:14.402 INFO:teuthology.orchestra.run.smithi173.stdout:Size : 8264254 2024-03-26T01:45:14.402 INFO:teuthology.orchestra.run.smithi173.stdout:License : GPLv3+ 2024-03-26T01:45:14.402 INFO:teuthology.orchestra.run.smithi173.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T01:45:14.402 INFO:teuthology.orchestra.run.smithi173.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T01:45:14.402 INFO:teuthology.orchestra.run.smithi173.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T01:45:14.402 INFO:teuthology.orchestra.run.smithi173.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T01:45:14.402 INFO:teuthology.orchestra.run.smithi173.stdout:Packager : builder@centos.org 2024-03-26T01:45:14.403 INFO:teuthology.orchestra.run.smithi173.stdout:Vendor : CentOS 2024-03-26T01:45:14.403 INFO:teuthology.orchestra.run.smithi173.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T01:45:14.403 INFO:teuthology.orchestra.run.smithi173.stdout:Summary : Support tools for GRUB. 2024-03-26T01:45:14.403 INFO:teuthology.orchestra.run.smithi173.stdout:Description : 2024-03-26T01:45:14.403 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:45:14.403 INFO:teuthology.orchestra.run.smithi173.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T01:45:14.403 INFO:teuthology.orchestra.run.smithi173.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T01:45:14.403 INFO:teuthology.orchestra.run.smithi173.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T01:45:14.403 INFO:teuthology.orchestra.run.smithi173.stdout:hardware devices. 2024-03-26T01:45:14.403 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:45:14.403 INFO:teuthology.orchestra.run.smithi173.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T01:45:14.404 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T01:45:14.404 INFO:teuthology.task.kernel:Updating grub on smithi173 to boot 5.14.0-432.el9.x86_64 2024-03-26T01:45:14.404 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T01:45:14.990 INFO:teuthology.orchestra.run.smithi173.stderr:Generating grub configuration file ... 2024-03-26T01:45:16.452 INFO:teuthology.orchestra.run.smithi173.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T01:45:16.485 INFO:teuthology.orchestra.run.smithi173.stderr:done 2024-03-26T01:45:16.488 DEBUG:teuthology.orchestra.run.smithi173:> mktemp 2024-03-26T01:45:16.503 INFO:teuthology.orchestra.run.smithi173.stdout:/tmp/tmp.gibwPQU6Tx 2024-03-26T01:45:16.503 DEBUG:teuthology.orchestra.run.smithi173:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.gibwPQU6Tx 2024-03-26T01:45:16.572 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0666 /tmp/tmp.gibwPQU6Tx 2024-03-26T01:45:16.715 DEBUG:teuthology.orchestra.remote:smithi173:/tmp/tmp.gibwPQU6Tx is 6KB 2024-03-26T01:45:16.764 DEBUG:teuthology.orchestra.run.smithi173:> rm -fr /tmp/tmp.gibwPQU6Tx 2024-03-26T01:45:16.779 DEBUG:teuthology.orchestra.run.smithi173:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T01:45:16.843 INFO:teuthology.orchestra.run.smithi173.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T01:45:16.843 INFO:teuthology.orchestra.run.smithi173.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T01:45:16.843 INFO:teuthology.orchestra.run.smithi173.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T01:45:16.843 INFO:teuthology.orchestra.run.smithi173.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T01:45:16.844 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T01:45:16.998 DEBUG:teuthology.orchestra.run.smithi173:> sudo shutdown -r now 2024-03-26T01:45:17.226 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:45:17.226 INFO:teuthology.orchestra.run.smithi072.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T01:45:17.226 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:45:17.271 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:45:17.680 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:45:17.680 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T01:45:17.680 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T01:45:17.681 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T01:45:18.079 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T01:45:18.080 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:45:18.080 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-26T01:45:18.080 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T01:45:18.080 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T01:45:18.080 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T01:45:18.080 INFO:teuthology.orchestra.run.smithi072.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T01:45:18.080 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:45:18.080 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:45:18.487 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T01:45:18.630 INFO:teuthology.orchestra.run.smithi072.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T01:45:18.630 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T01:45:18.630 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T01:45:18.630 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T01:45:18.631 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T01:45:18.631 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-26T01:45:18.631 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi072.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T01:45:18.631 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T01:45:18.710 INFO:teuthology.orchestra.run.smithi072.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T01:45:18.711 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T01:45:18.711 DEBUG:teuthology.orchestra.run.smithi072:> sudo rpm -qi grub2-tools 2024-03-26T01:45:18.750 INFO:teuthology.orchestra.run.smithi072.stdout:Name : grub2-tools 2024-03-26T01:45:18.750 INFO:teuthology.orchestra.run.smithi072.stdout:Epoch : 1 2024-03-26T01:45:18.750 INFO:teuthology.orchestra.run.smithi072.stdout:Version : 2.06 2024-03-26T01:45:18.750 INFO:teuthology.orchestra.run.smithi072.stdout:Release : 68.el9 2024-03-26T01:45:18.750 INFO:teuthology.orchestra.run.smithi072.stdout:Architecture: x86_64 2024-03-26T01:45:18.750 INFO:teuthology.orchestra.run.smithi072.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T01:45:18.750 INFO:teuthology.orchestra.run.smithi072.stdout:Group : Unspecified 2024-03-26T01:45:18.750 INFO:teuthology.orchestra.run.smithi072.stdout:Size : 8264254 2024-03-26T01:45:18.750 INFO:teuthology.orchestra.run.smithi072.stdout:License : GPLv3+ 2024-03-26T01:45:18.750 INFO:teuthology.orchestra.run.smithi072.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T01:45:18.750 INFO:teuthology.orchestra.run.smithi072.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T01:45:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T01:45:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T01:45:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:Packager : builder@centos.org 2024-03-26T01:45:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:Vendor : CentOS 2024-03-26T01:45:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T01:45:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:Summary : Support tools for GRUB. 2024-03-26T01:45:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:Description : 2024-03-26T01:45:18.751 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:45:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T01:45:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T01:45:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T01:45:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:hardware devices. 2024-03-26T01:45:18.751 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:45:18.751 INFO:teuthology.orchestra.run.smithi072.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T01:45:18.753 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T01:45:18.753 INFO:teuthology.task.kernel:Updating grub on smithi072 to boot 5.14.0-432.el9.x86_64 2024-03-26T01:45:18.753 DEBUG:teuthology.orchestra.run.smithi072:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T01:45:19.403 INFO:teuthology.orchestra.run.smithi072.stderr:Generating grub configuration file ... 2024-03-26T01:45:20.921 INFO:teuthology.orchestra.run.smithi072.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T01:45:20.947 INFO:teuthology.orchestra.run.smithi072.stderr:done 2024-03-26T01:45:20.949 DEBUG:teuthology.orchestra.run.smithi072:> mktemp 2024-03-26T01:45:20.984 INFO:teuthology.orchestra.run.smithi072.stdout:/tmp/tmp.8XKUFuUH3U 2024-03-26T01:45:20.984 DEBUG:teuthology.orchestra.run.smithi072:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.8XKUFuUH3U 2024-03-26T01:45:21.053 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0666 /tmp/tmp.8XKUFuUH3U 2024-03-26T01:45:21.195 DEBUG:teuthology.orchestra.remote:smithi072:/tmp/tmp.8XKUFuUH3U is 6KB 2024-03-26T01:45:21.243 DEBUG:teuthology.orchestra.run.smithi072:> rm -fr /tmp/tmp.8XKUFuUH3U 2024-03-26T01:45:21.257 DEBUG:teuthology.orchestra.run.smithi072:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T01:45:21.322 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T01:45:21.323 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T01:45:21.323 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T01:45:21.323 INFO:teuthology.orchestra.run.smithi072.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T01:45:21.324 DEBUG:teuthology.orchestra.run.smithi072:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T01:45:21.474 DEBUG:teuthology.orchestra.run.smithi072:> sudo shutdown -r now 2024-03-26T01:45:47.026 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-26T01:45:47.027 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-26T01:45:47.028 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:45:51.482 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-26T01:45:51.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-26T01:45:51.483 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:46:05.471 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-03-26T01:46:09.822 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.72 2024-03-26T01:46:14.476 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-26T01:46:14.477 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:46:17.534 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-03-26T01:46:18.824 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-26T01:46:18.825 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:46:21.886 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.72 2024-03-26T01:46:29.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-26T01:46:29.544 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:46:33.892 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-26T01:46:33.893 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:47:01.369 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-03-26T01:47:01.815 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-26T01:47:01.815 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-26T01:47:01.816 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-03-26T01:47:01.829 INFO:teuthology.orchestra.run.smithi173.stdout:5.14.0-432.el9.x86_64 2024-03-26T01:47:01.830 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T01:47:01.830 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T01:47:01.830 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T01:47:02.831 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-26T01:47:02.831 DEBUG:teuthology.orchestra.run.smithi173:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T01:47:02.926 INFO:teuthology.orchestra.run.smithi173.stdout:ttyS1 2024-03-26T01:47:02.959 DEBUG:teuthology.parallel:result is None 2024-03-26T01:47:05.797 DEBUG:teuthology.orchestra.run.smithi072:> true 2024-03-26T01:47:06.351 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-26T01:47:06.351 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-26T01:47:06.351 DEBUG:teuthology.orchestra.run.smithi072:> uname -r 2024-03-26T01:47:06.365 INFO:teuthology.orchestra.run.smithi072.stdout:5.14.0-432.el9.x86_64 2024-03-26T01:47:06.365 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T01:47:06.365 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T01:47:06.365 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T01:47:07.366 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-26T01:47:07.366 DEBUG:teuthology.orchestra.run.smithi072:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T01:47:07.543 INFO:teuthology.orchestra.run.smithi072.stdout:ttyS1 2024-03-26T01:47:07.598 DEBUG:teuthology.parallel:result is None 2024-03-26T01:47:07.599 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-26T01:47:07.606 INFO:teuthology.task.internal:Creating test directory... 2024-03-26T01:47:07.606 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T01:47:07.608 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T01:47:07.626 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-26T01:47:07.633 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-26T01:47:07.643 INFO:teuthology.task.internal:Creating archive directory... 2024-03-26T01:47:07.643 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T01:47:07.663 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T01:47:07.835 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-26T01:47:07.845 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-26T01:47:07.846 DEBUG:teuthology.orchestra.run.smithi072:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T01:47:07.848 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T01:47:07.881 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T01:47:07.883 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T01:47:07.894 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T01:47:07.897 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T01:47:07.899 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-26T01:47:07.909 INFO:teuthology.task.internal:Configuring sudo... 2024-03-26T01:47:07.909 DEBUG:teuthology.orchestra.run.smithi072:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T01:47:07.938 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T01:47:07.969 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-26T01:47:07.978 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-26T01:47:07.979 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T01:47:08.006 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T01:47:08.026 DEBUG:teuthology.orchestra.run.smithi072:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T01:47:08.103 DEBUG:teuthology.orchestra.run.smithi072:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T01:47:08.167 DEBUG:teuthology.orchestra.run.smithi072:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T01:47:08.249 DEBUG:teuthology.orchestra.run.smithi072:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T01:47:08.299 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-26T01:47:08.299 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T01:47:08.376 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T01:47:08.416 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T01:47:08.468 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T01:47:08.545 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T01:47:08.596 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-26T01:47:08.596 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T01:47:08.665 DEBUG:teuthology.orchestra.run.smithi072:> sudo service rsyslog restart 2024-03-26T01:47:08.667 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2024-03-26T01:47:08.711 INFO:teuthology.orchestra.run.smithi072.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T01:47:08.763 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T01:47:09.097 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-26T01:47:09.106 INFO:teuthology.task.internal:Starting timer... 2024-03-26T01:47:09.106 INFO:teuthology.run_tasks:Running task pcp... 2024-03-26T01:47:09.126 INFO:teuthology.run_tasks:Running task selinux... 2024-03-26T01:47:09.134 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-26T01:47:09.135 DEBUG:teuthology.orchestra.run.smithi072:> sudo service auditd rotate 2024-03-26T01:47:09.238 INFO:teuthology.orchestra.run.smithi072.stdout:Rotating logs: 2024-03-26T01:47:09.240 DEBUG:teuthology.orchestra.run.smithi173:> sudo service auditd rotate 2024-03-26T01:47:09.286 INFO:teuthology.orchestra.run.smithi173.stdout:Rotating logs: 2024-03-26T01:47:09.288 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-26T01:47:09.288 DEBUG:teuthology.orchestra.run.smithi072:> /usr/sbin/getenforce 2024-03-26T01:47:09.324 INFO:teuthology.orchestra.run.smithi072.stdout:Permissive 2024-03-26T01:47:09.324 DEBUG:teuthology.orchestra.run.smithi173:> /usr/sbin/getenforce 2024-03-26T01:47:09.343 INFO:teuthology.orchestra.run.smithi173.stdout:Permissive 2024-03-26T01:47:09.344 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi072.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi173.front.sepia.ceph.com': 'permissive'} 2024-03-26T01:47:09.344 DEBUG:teuthology.orchestra.run.smithi072:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T01:47:09.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T01:47:09.397 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T01:47:09.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T01:47:09.425 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-26T01:47:09.425 DEBUG:teuthology.orchestra.run.smithi072:> sudo /usr/sbin/setenforce permissive 2024-03-26T01:47:09.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo /usr/sbin/setenforce permissive 2024-03-26T01:47:09.512 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-26T01:47:09.522 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-26T01:47:09.527 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-26T01:47:09.653 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-26T01:47:09.668 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-26T01:47:09.670 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi072.front.sepia.ceph.com,smithi173.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-26T01:51:41.078 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi072.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com')] 2024-03-26T01:51:41.079 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-26T01:51:41.081 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi072.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:51:41.175 DEBUG:teuthology.orchestra.run.smithi072:> true 2024-03-26T01:51:41.252 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi072.front.sepia.ceph.com' 2024-03-26T01:51:41.253 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-26T01:51:41.253 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T01:51:41.340 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-03-26T01:51:41.430 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-03-26T01:51:41.430 INFO:teuthology.run_tasks:Running task clock... 2024-03-26T01:51:41.441 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-26T01:51:41.442 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T01:51:41.442 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T01:51:41.445 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T01:51:41.445 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T01:51:41.474 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T01:51:41.491 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T01:51:41.501 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T01:51:41.518 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T01:51:41.520 INFO:teuthology.orchestra.run.smithi072.stderr:sudo: ntpd: command not found 2024-03-26T01:51:41.534 INFO:teuthology.orchestra.run.smithi072.stdout:506 Cannot talk to daemon 2024-03-26T01:51:41.547 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: ntpd: command not found 2024-03-26T01:51:41.550 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T01:51:41.561 INFO:teuthology.orchestra.run.smithi173.stdout:506 Cannot talk to daemon 2024-03-26T01:51:41.566 INFO:teuthology.orchestra.run.smithi072.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T01:51:41.577 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T01:51:41.593 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T01:51:41.621 INFO:teuthology.orchestra.run.smithi072.stderr:bash: line 1: ntpq: command not found 2024-03-26T01:51:41.624 INFO:teuthology.orchestra.run.smithi072.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T01:51:41.624 INFO:teuthology.orchestra.run.smithi072.stdout:=============================================================================== 2024-03-26T01:51:41.624 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:51:41.625 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:51:41.625 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:51:41.625 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:51:41.655 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-03-26T01:51:41.658 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T01:51:41.658 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-03-26T01:51:41.658 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:51:41.658 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:51:41.658 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:51:41.658 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T01:51:41.659 INFO:teuthology.run_tasks:Running task pexec... 2024-03-26T01:51:41.669 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-26T01:51:41.669 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi072.front.sepia.ceph.com 2024-03-26T01:51:41.669 DEBUG:teuthology.orchestra.run.smithi072:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-26T01:51:41.670 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi173.front.sepia.ceph.com 2024-03-26T01:51:41.670 DEBUG:teuthology.orchestra.run.smithi173:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-26T01:51:42.245 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:04 ago on Tue 26 Mar 2024 01:50:38 AM UTC. 2024-03-26T01:51:42.284 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:07 ago on Tue 26 Mar 2024 01:50:35 AM UTC. 2024-03-26T01:51:42.390 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:51:42.408 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:51:42.408 INFO:teuthology.orchestra.run.smithi072.stdout: Package Architecture Version Repository Size 2024-03-26T01:51:42.408 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:51:42.408 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-26T01:51:42.409 INFO:teuthology.orchestra.run.smithi072.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-26T01:51:42.409 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:51:42.409 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-26T01:51:42.409 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:51:42.409 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2024-03-26T01:51:42.409 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:51:42.409 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 3.1 M 2024-03-26T01:51:42.409 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 9.8 M 2024-03-26T01:51:42.409 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-26T01:51:42.430 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:51:42.435 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:51:42.436 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-03-26T01:51:42.436 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:51:42.436 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-26T01:51:42.436 INFO:teuthology.orchestra.run.smithi173.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-26T01:51:42.436 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:51:42.436 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-26T01:51:42.436 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:51:42.436 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-26T01:51:42.436 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:51:42.437 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 3.1 M 2024-03-26T01:51:42.437 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 9.8 M 2024-03-26T01:51:42.437 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-26T01:51:43.039 INFO:teuthology.orchestra.run.smithi173.stdout:runc-1.1.12-2.el9.x86_64.rpm 9.3 MB/s | 3.1 MB 00:00 2024-03-26T01:51:43.040 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:51:43.040 INFO:teuthology.orchestra.run.smithi173.stdout:Total 5.2 MB/s | 3.1 MB 00:00 2024-03-26T01:51:43.076 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-26T01:51:43.081 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-26T01:51:43.081 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-26T01:51:43.161 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-26T01:51:43.161 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-26T01:51:43.382 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-26T01:51:43.469 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T01:51:43.509 INFO:teuthology.orchestra.run.smithi072.stdout:runc-1.1.12-2.el9.x86_64.rpm 3.0 MB/s | 3.1 MB 00:01 2024-03-26T01:51:43.509 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:51:43.510 INFO:teuthology.orchestra.run.smithi072.stdout:Total 2.8 MB/s | 3.1 MB 00:01 2024-03-26T01:51:43.547 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-26T01:51:43.553 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-26T01:51:43.553 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-26T01:51:43.635 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-26T01:51:43.635 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-26T01:51:43.857 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-26T01:51:43.947 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T01:51:43.954 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T01:51:44.264 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T01:51:44.264 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:51:44.265 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-26T01:51:44.265 INFO:teuthology.orchestra.run.smithi173.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-26T01:51:44.265 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:51:44.265 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:51:44.529 DEBUG:teuthology.parallel:result is None 2024-03-26T01:51:44.685 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T01:51:44.932 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-26T01:51:44.932 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:51:44.932 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-26T01:51:44.932 INFO:teuthology.orchestra.run.smithi072.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-26T01:51:44.932 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:51:44.932 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:51:45.177 DEBUG:teuthology.parallel:result is None 2024-03-26T01:51:45.177 INFO:teuthology.run_tasks:Running task install... 2024-03-26T01:51:45.186 DEBUG:teuthology.task.install:project ceph 2024-03-26T01:51:45.187 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a'}, 'extra_packages': ['cephadm']} 2024-03-26T01:51:45.187 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a'} 2024-03-26T01:51:45.187 INFO:teuthology.task.install:Using flavor: default 2024-03-26T01:51:45.193 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-26T01:51:45.193 INFO:teuthology.task.install:extra packages: [] 2024-03-26T01:51:45.193 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a', 'tag': None, 'wait_for_package': False} 2024-03-26T01:51:45.194 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T01:51:45.196 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a', 'tag': None, 'wait_for_package': False} 2024-03-26T01:51:45.196 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T01:51:45.344 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/ 2024-03-26T01:51:45.344 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1775.g5e70124b 2024-03-26T01:51:45.345 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/ 2024-03-26T01:51:45.346 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1775.g5e70124b 2024-03-26T01:51:45.458 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-26T01:51:45.458 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-26T01:51:45.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-26T01:51:45.487 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-26T01:51:45.488 DEBUG:teuthology.orchestra.run.smithi173:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5e70124b46c2a671efcaae4690cd9ea72670710a/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-26T01:51:45.529 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-26T01:51:45.529 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-26T01:51:45.529 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-26T01:51:45.556 DEBUG:teuthology.orchestra.run.smithi173:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-26T01:51:45.557 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-26T01:51:45.557 DEBUG:teuthology.orchestra.run.smithi072:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5e70124b46c2a671efcaae4690cd9ea72670710a/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-26T01:51:45.626 DEBUG:teuthology.orchestra.run.smithi072:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-26T01:51:45.635 DEBUG:teuthology.orchestra.run.smithi173:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-26T01:51:45.706 INFO:teuthology.orchestra.run.smithi173.stdout:check_obsoletes = 1 2024-03-26T01:51:45.706 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum clean all 2024-03-26T01:51:45.707 DEBUG:teuthology.orchestra.run.smithi072:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-26T01:51:45.783 INFO:teuthology.orchestra.run.smithi072.stdout:check_obsoletes = 1 2024-03-26T01:51:45.785 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum clean all 2024-03-26T01:51:45.950 INFO:teuthology.orchestra.run.smithi173.stdout:74 files removed 2024-03-26T01:51:45.977 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-radosgw 2024-03-26T01:51:46.027 INFO:teuthology.orchestra.run.smithi072.stdout:74 files removed 2024-03-26T01:51:46.054 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-radosgw 2024-03-26T01:51:46.774 INFO:teuthology.orchestra.run.smithi072.stdout:ceph packages for x86_64 199 kB/s | 83 kB 00:00 2024-03-26T01:51:46.847 INFO:teuthology.orchestra.run.smithi173.stdout:ceph packages for x86_64 146 kB/s | 83 kB 00:00 2024-03-26T01:51:47.284 INFO:teuthology.orchestra.run.smithi072.stdout:ceph noarch packages 22 kB/s | 10 kB 00:00 2024-03-26T01:51:47.331 INFO:teuthology.orchestra.run.smithi173.stdout:ceph noarch packages 23 kB/s | 10 kB 00:00 2024-03-26T01:51:47.652 INFO:teuthology.orchestra.run.smithi072.stdout:ceph source packages 5.8 kB/s | 1.9 kB 00:00 2024-03-26T01:51:47.682 INFO:teuthology.orchestra.run.smithi173.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-03-26T01:51:48.190 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-26T01:51:48.443 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - AppStream 9.3 MB/s | 6.7 MB 00:00 2024-03-26T01:51:49.600 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-26T01:51:49.968 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS-9 - BaseOS 4.8 MB/s | 2.0 MB 00:00 2024-03-26T01:51:50.817 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-26T01:51:52.141 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - BaseOS 3.6 MB/s | 8.0 MB 00:02 2024-03-26T01:51:54.102 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-26T01:51:55.252 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 5.6 MB/s | 19 MB 00:03 2024-03-26T01:52:00.139 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 6.1 MB 00:02 2024-03-26T01:52:01.013 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.1 MB 00:02 2024-03-26T01:52:01.599 INFO:teuthology.orchestra.run.smithi072.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 16 kB 00:00 2024-03-26T01:52:02.496 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 16 kB 00:00 2024-03-26T01:52:02.627 INFO:teuthology.orchestra.run.smithi072.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-26T01:52:07.496 INFO:teuthology.orchestra.run.smithi072.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-26T01:52:10.040 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:52:10.042 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:52:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-03-26T01:52:10.042 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:52:10.042 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-26T01:52:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-radosgw x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 9.8 M 2024-03-26T01:52:10.042 INFO:teuthology.orchestra.run.smithi072.stdout:Upgrading: 2024-03-26T01:52:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: librados2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.4 M 2024-03-26T01:52:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: librbd1 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.1 M 2024-03-26T01:52:10.042 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-03-26T01:52:10.042 INFO:teuthology.orchestra.run.smithi072.stdout: boost-program-options 2024-03-26T01:52:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-26T01:52:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-base x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 5.3 M 2024-03-26T01:52:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-common x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 20 M 2024-03-26T01:52:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-selinux x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 25 k 2024-03-26T01:52:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-26T01:52:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-26T01:52:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-26T01:52:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 744 k 2024-03-26T01:52:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-26T01:52:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-26T01:52:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-26T01:52:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: libradosstriper1 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 498 k 2024-03-26T01:52:10.043 INFO:teuthology.orchestra.run.smithi072.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-26T01:52:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: librgw2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 5.1 M 2024-03-26T01:52:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-26T01:52:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-26T01:52:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-26T01:52:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-argparse 2024-03-26T01:52:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 45 k 2024-03-26T01:52:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-common x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 131 k 2024-03-26T01:52:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cephfs x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 162 k 2024-03-26T01:52:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-26T01:52:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rados x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 321 k 2024-03-26T01:52:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rbd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 298 k 2024-03-26T01:52:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rgw x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 99 k 2024-03-26T01:52:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-26T01:52:10.044 INFO:teuthology.orchestra.run.smithi072.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-26T01:52:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-26T01:52:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-26T01:52:10.045 INFO:teuthology.orchestra.run.smithi072.stdout:Installing weak dependencies: 2024-03-26T01:52:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-26T01:52:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:52:10.045 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-26T01:52:10.045 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:52:10.045 INFO:teuthology.orchestra.run.smithi072.stdout:Install 30 Packages 2024-03-26T01:52:10.045 INFO:teuthology.orchestra.run.smithi072.stdout:Upgrade 2 Packages 2024-03-26T01:52:10.045 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:52:10.046 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 58 M 2024-03-26T01:52:10.046 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-26T01:52:11.382 INFO:teuthology.orchestra.run.smithi072.stdout:(1/32): ceph-base-19.0.0-1775.g5e70124b.el9.x86 6.7 MB/s | 5.3 MB 00:00 2024-03-26T01:52:11.432 INFO:teuthology.orchestra.run.smithi072.stdout:(2/32): ceph-selinux-19.0.0-1775.g5e70124b.el9. 502 kB/s | 25 kB 00:00 2024-03-26T01:52:11.549 INFO:teuthology.orchestra.run.smithi072.stdout:(3/32): libcephfs2-19.0.0-1775.g5e70124b.el9.x8 6.2 MB/s | 744 kB 00:00 2024-03-26T01:52:11.650 INFO:teuthology.orchestra.run.smithi072.stdout:(4/32): libradosstriper1-19.0.0-1775.g5e70124b. 4.9 MB/s | 498 kB 00:00 2024-03-26T01:52:12.209 INFO:teuthology.orchestra.run.smithi072.stdout:(5/32): ceph-common-19.0.0-1775.g5e70124b.el9.x 13 MB/s | 20 MB 00:01 2024-03-26T01:52:12.326 INFO:teuthology.orchestra.run.smithi072.stdout:(6/32): librgw2-19.0.0-1775.g5e70124b.el9.x86_6 7.6 MB/s | 5.1 MB 00:00 2024-03-26T01:52:12.352 INFO:teuthology.orchestra.run.smithi072.stdout:(7/32): python3-ceph-argparse-19.0.0-1775.g5e70 317 kB/s | 45 kB 00:00 2024-03-26T01:52:12.385 INFO:teuthology.orchestra.run.smithi072.stdout:(8/32): python3-ceph-common-19.0.0-1775.g5e7012 2.2 MB/s | 131 kB 00:00 2024-03-26T01:52:12.411 INFO:teuthology.orchestra.run.smithi072.stdout:(9/32): python3-cephfs-19.0.0-1775.g5e70124b.el 2.7 MB/s | 162 kB 00:00 2024-03-26T01:52:12.446 INFO:teuthology.orchestra.run.smithi072.stdout:(10/32): python3-rados-19.0.0-1775.g5e70124b.el 5.2 MB/s | 321 kB 00:00 2024-03-26T01:52:12.480 INFO:teuthology.orchestra.run.smithi072.stdout:(11/32): python3-rbd-19.0.0-1775.g5e70124b.el9. 4.2 MB/s | 298 kB 00:00 2024-03-26T01:52:12.506 INFO:teuthology.orchestra.run.smithi072.stdout:(12/32): python3-rgw-19.0.0-1775.g5e70124b.el9. 1.6 MB/s | 99 kB 00:00 2024-03-26T01:52:12.623 INFO:teuthology.orchestra.run.smithi072.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 743 kB/s | 106 kB 00:00 2024-03-26T01:52:12.648 INFO:teuthology.orchestra.run.smithi072.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-03-26T01:52:12.666 INFO:teuthology.orchestra.run.smithi072.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-26T01:52:12.725 INFO:teuthology.orchestra.run.smithi072.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.5 MB/s | 661 kB 00:00 2024-03-26T01:52:12.758 INFO:teuthology.orchestra.run.smithi072.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-26T01:52:12.776 INFO:teuthology.orchestra.run.smithi072.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 816 kB/s | 41 kB 00:00 2024-03-26T01:52:13.043 INFO:teuthology.orchestra.run.smithi072.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 134 kB/s | 36 kB 00:00 2024-03-26T01:52:13.227 INFO:teuthology.orchestra.run.smithi072.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 258 kB/s | 47 kB 00:00 2024-03-26T01:52:13.394 INFO:teuthology.orchestra.run.smithi072.stdout:(21/32): ceph-radosgw-19.0.0-1775.g5e70124b.el9 3.5 MB/s | 9.8 MB 00:02 2024-03-26T01:52:13.428 INFO:teuthology.orchestra.run.smithi072.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 330 kB/s | 221 kB 00:00 2024-03-26T01:52:13.662 INFO:teuthology.orchestra.run.smithi072.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 111 kB/s | 26 kB 00:00 2024-03-26T01:52:13.779 INFO:teuthology.orchestra.run.smithi072.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 559 kB/s | 309 kB 00:00 2024-03-26T01:52:13.831 INFO:teuthology.orchestra.run.smithi072.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 287 kB/s | 49 kB 00:00 2024-03-26T01:52:13.882 INFO:teuthology.orchestra.run.smithi072.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 659 kB/s | 67 kB 00:00 2024-03-26T01:52:13.982 INFO:teuthology.orchestra.run.smithi072.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-26T01:52:14.183 INFO:teuthology.orchestra.run.smithi072.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.6 MB/s | 4.4 MB 00:00 2024-03-26T01:52:14.216 INFO:teuthology.orchestra.run.smithi072.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.1 MB/s | 838 kB 00:00 2024-03-26T01:52:14.300 INFO:teuthology.orchestra.run.smithi072.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 5.0 MB/s | 1.6 MB 00:00 2024-03-26T01:52:14.584 INFO:teuthology.orchestra.run.smithi072.stdout:(31/32): librados2-19.0.0-1775.g5e70124b.el9.x8 8.4 MB/s | 3.4 MB 00:00 2024-03-26T01:52:14.659 INFO:teuthology.orchestra.run.smithi072.stdout:(32/32): librbd1-19.0.0-1775.g5e70124b.el9.x86_ 7.1 MB/s | 3.1 MB 00:00 2024-03-26T01:52:14.662 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:52:14.663 INFO:teuthology.orchestra.run.smithi072.stdout:Total 13 MB/s | 58 MB 00:04 2024-03-26T01:52:14.696 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-26T01:52:14.734 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-26T01:52:14.734 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-26T01:52:15.218 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-26T01:52:15.222 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-26T01:52:16.655 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-26T01:52:16.750 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-26T01:52:16.782 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-26T01:52:16.818 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-26T01:52:16.850 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T01:52:16.865 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T01:52:17.018 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T01:52:17.074 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T01:52:17.417 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-26T01:52:17.543 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T01:52:17.649 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T01:52:17.755 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 8/34 2024-03-26T01:52:17.787 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 9/34 2024-03-26T01:52:17.850 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 9/34 2024-03-26T01:52:17.888 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-26T01:52:17.940 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9. 11/34 2024-03-26T01:52:18.009 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 12/34 2024-03-26T01:52:18.026 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 13/34 2024-03-26T01:52:18.083 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 13/34 2024-03-26T01:52:18.117 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-26T01:52:18.226 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-26T01:52:18.270 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-26T01:52:18.310 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-26T01:52:18.344 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-26T01:52:18.374 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-26T01:52:18.409 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-26T01:52:18.718 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-26T01:52:18.748 INFO:teuthology.orchestra.run.smithi072.stdout: Upgrading : librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 22/34 2024-03-26T01:52:18.880 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 22/34 2024-03-26T01:52:18.912 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 23/34 2024-03-26T01:52:18.961 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-26T01:52:19.416 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-26T01:52:19.455 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 26/34 2024-03-26T01:52:19.518 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 26/34 2024-03-26T01:52:19.595 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 27/34 2024-03-26T01:52:19.625 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x8 28/34 2024-03-26T01:52:20.228 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 1.2 MB/s | 21 MB 00:17 2024-03-26T01:52:21.873 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T01:52:21.920 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T01:52:22.463 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T01:52:22.512 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 30/34 2024-03-26T01:52:22.606 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 30/34 2024-03-26T01:52:22.606 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-26T01:52:22.606 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-26T01:52:22.606 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:52:22.673 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T01:52:25.162 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-26T01:52:27.620 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:52:27.622 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:52:27.623 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-26T01:52:27.623 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:52:27.623 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-26T01:52:27.623 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 9.8 M 2024-03-26T01:52:27.623 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrading: 2024-03-26T01:52:27.623 INFO:teuthology.orchestra.run.smithi173.stdout: librados2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.4 M 2024-03-26T01:52:27.623 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.1 M 2024-03-26T01:52:27.623 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-26T01:52:27.623 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options 2024-03-26T01:52:27.623 INFO:teuthology.orchestra.run.smithi173.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-26T01:52:27.623 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 5.3 M 2024-03-26T01:52:27.623 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 20 M 2024-03-26T01:52:27.623 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 25 k 2024-03-26T01:52:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-26T01:52:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-26T01:52:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-26T01:52:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 744 k 2024-03-26T01:52:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-26T01:52:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-26T01:52:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-26T01:52:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 498 k 2024-03-26T01:52:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-26T01:52:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 5.1 M 2024-03-26T01:52:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-26T01:52:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-26T01:52:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-26T01:52:27.624 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse 2024-03-26T01:52:27.625 INFO:teuthology.orchestra.run.smithi173.stdout: x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 45 k 2024-03-26T01:52:27.625 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 131 k 2024-03-26T01:52:27.625 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 162 k 2024-03-26T01:52:27.625 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-26T01:52:27.625 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 321 k 2024-03-26T01:52:27.625 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 298 k 2024-03-26T01:52:27.625 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 99 k 2024-03-26T01:52:27.625 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-26T01:52:27.625 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-26T01:52:27.625 INFO:teuthology.orchestra.run.smithi173.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-26T01:52:27.625 INFO:teuthology.orchestra.run.smithi173.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-26T01:52:27.625 INFO:teuthology.orchestra.run.smithi173.stdout:Installing weak dependencies: 2024-03-26T01:52:27.625 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-26T01:52:27.625 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:52:27.626 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-26T01:52:27.626 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:52:27.626 INFO:teuthology.orchestra.run.smithi173.stdout:Install 30 Packages 2024-03-26T01:52:27.626 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrade 2 Packages 2024-03-26T01:52:27.626 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:52:27.626 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 58 M 2024-03-26T01:52:27.626 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-26T01:52:28.655 INFO:teuthology.orchestra.run.smithi173.stdout:(1/32): ceph-base-19.0.0-1775.g5e70124b.el9.x86 8.4 MB/s | 5.3 MB 00:00 2024-03-26T01:52:28.698 INFO:teuthology.orchestra.run.smithi173.stdout:(2/32): ceph-selinux-19.0.0-1775.g5e70124b.el9. 605 kB/s | 25 kB 00:00 2024-03-26T01:52:28.797 INFO:teuthology.orchestra.run.smithi173.stdout:(3/32): libcephfs2-19.0.0-1775.g5e70124b.el9.x8 7.3 MB/s | 744 kB 00:00 2024-03-26T01:52:28.880 INFO:teuthology.orchestra.run.smithi173.stdout:(4/32): libradosstriper1-19.0.0-1775.g5e70124b. 5.8 MB/s | 498 kB 00:00 2024-03-26T01:52:29.397 INFO:teuthology.orchestra.run.smithi173.stdout:(5/32): librgw2-19.0.0-1775.g5e70124b.el9.x86_6 9.9 MB/s | 5.1 MB 00:00 2024-03-26T01:52:29.439 INFO:teuthology.orchestra.run.smithi173.stdout:(6/32): python3-ceph-argparse-19.0.0-1775.g5e70 1.1 MB/s | 45 kB 00:00 2024-03-26T01:52:29.548 INFO:teuthology.orchestra.run.smithi173.stdout:(7/32): ceph-radosgw-19.0.0-1775.g5e70124b.el9. 6.4 MB/s | 9.8 MB 00:01 2024-03-26T01:52:29.573 INFO:teuthology.orchestra.run.smithi173.stdout:(8/32): python3-ceph-common-19.0.0-1775.g5e7012 984 kB/s | 131 kB 00:00 2024-03-26T01:52:29.632 INFO:teuthology.orchestra.run.smithi173.stdout:(9/32): python3-rados-19.0.0-1775.g5e70124b.el9 5.4 MB/s | 321 kB 00:00 2024-03-26T01:52:29.657 INFO:teuthology.orchestra.run.smithi173.stdout:(10/32): python3-cephfs-19.0.0-1775.g5e70124b.e 1.5 MB/s | 162 kB 00:00 2024-03-26T01:52:29.690 INFO:teuthology.orchestra.run.smithi173.stdout:(11/32): python3-rbd-19.0.0-1775.g5e70124b.el9. 5.0 MB/s | 298 kB 00:00 2024-03-26T01:52:29.749 INFO:teuthology.orchestra.run.smithi173.stdout:(12/32): python3-rgw-19.0.0-1775.g5e70124b.el9. 1.1 MB/s | 99 kB 00:00 2024-03-26T01:52:29.892 INFO:teuthology.orchestra.run.smithi173.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 528 kB/s | 106 kB 00:00 2024-03-26T01:52:29.925 INFO:teuthology.orchestra.run.smithi173.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 914 kB/s | 159 kB 00:00 2024-03-26T01:52:29.950 INFO:teuthology.orchestra.run.smithi173.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 757 kB/s | 44 kB 00:00 2024-03-26T01:52:30.000 INFO:teuthology.orchestra.run.smithi173.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-26T01:52:30.025 INFO:teuthology.orchestra.run.smithi173.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-26T01:52:30.043 INFO:teuthology.orchestra.run.smithi173.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 990 kB/s | 41 kB 00:00 2024-03-26T01:52:30.385 INFO:teuthology.orchestra.run.smithi173.stdout:(19/32): ceph-common-19.0.0-1775.g5e70124b.el9. 8.6 MB/s | 20 MB 00:02 2024-03-26T01:52:30.569 INFO:teuthology.orchestra.run.smithi173.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 68 kB/s | 36 kB 00:00 2024-03-26T01:52:30.727 INFO:teuthology.orchestra.run.smithi173.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 139 kB/s | 47 kB 00:00 2024-03-26T01:52:30.886 INFO:teuthology.orchestra.run.smithi173.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 257 kB/s | 221 kB 00:00 2024-03-26T01:52:31.694 INFO:teuthology.orchestra.run.smithi173.stdout:(23/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.6 MB/s | 4.4 MB 00:00 2024-03-26T01:52:31.753 INFO:teuthology.orchestra.run.smithi173.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 834 kB/s | 49 kB 00:00 2024-03-26T01:52:31.820 INFO:teuthology.orchestra.run.smithi173.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-26T01:52:31.954 INFO:teuthology.orchestra.run.smithi173.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 6.1 MB/s | 838 kB 00:00 2024-03-26T01:52:32.021 INFO:teuthology.orchestra.run.smithi173.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-26T01:52:32.212 INFO:teuthology.orchestra.run.smithi173.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 8.3 MB/s | 1.6 MB 00:00 2024-03-26T01:52:32.496 INFO:teuthology.orchestra.run.smithi173.stdout:(29/32): librados2-19.0.0-1775.g5e70124b.el9.x8 12 MB/s | 3.4 MB 00:00 2024-03-26T01:52:32.672 INFO:teuthology.orchestra.run.smithi173.stdout:(30/32): librbd1-19.0.0-1775.g5e70124b.el9.x86_ 18 MB/s | 3.1 MB 00:00 2024-03-26T01:52:34.012 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T01:52:34.012 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /sys 2024-03-26T01:52:34.012 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /proc 2024-03-26T01:52:34.013 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /mnt 2024-03-26T01:52:34.013 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /var/tmp 2024-03-26T01:52:34.013 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /home 2024-03-26T01:52:34.013 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /root 2024-03-26T01:52:34.013 INFO:teuthology.orchestra.run.smithi072.stdout:skipping the directory /tmp 2024-03-26T01:52:34.013 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:52:34.371 INFO:teuthology.orchestra.run.smithi173.stdout:(31/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 7.4 kB/s | 26 kB 00:03 2024-03-26T01:52:34.505 INFO:teuthology.orchestra.run.smithi173.stdout:(32/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 78 kB/s | 309 kB 00:03 2024-03-26T01:52:34.508 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:52:34.508 INFO:teuthology.orchestra.run.smithi173.stdout:Total 8.5 MB/s | 58 MB 00:06 2024-03-26T01:52:34.543 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-26T01:52:34.581 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-26T01:52:34.581 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-26T01:52:34.957 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 32/34 2024-03-26T01:52:34.981 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 32/34 2024-03-26T01:52:34.981 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T01:52:34.981 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-26T01:52:34.981 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T01:52:34.981 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T01:52:34.981 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:52:34.983 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T01:52:35.061 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-26T01:52:35.061 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-26T01:52:35.078 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T01:52:35.078 INFO:teuthology.orchestra.run.smithi072.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T01:52:36.520 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T01:52:36.520 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 1/34 2024-03-26T01:52:36.520 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2/34 2024-03-26T01:52:36.520 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 3/34 2024-03-26T01:52:36.520 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 4/34 2024-03-26T01:52:36.520 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 5/34 2024-03-26T01:52:36.520 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 6/34 2024-03-26T01:52:36.521 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T01:52:36.521 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9. 8/34 2024-03-26T01:52:36.521 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x8 9/34 2024-03-26T01:52:36.521 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 10/34 2024-03-26T01:52:36.521 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 11/34 2024-03-26T01:52:36.521 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 12/34 2024-03-26T01:52:36.521 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 13/34 2024-03-26T01:52:36.521 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-26T01:52:36.521 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-26T01:52:36.521 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-26T01:52:36.521 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-26T01:52:36.521 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-26T01:52:36.521 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-26T01:52:36.522 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-26T01:52:36.522 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-26T01:52:36.522 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-26T01:52:36.522 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-26T01:52:36.522 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-26T01:52:36.522 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-26T01:52:36.522 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-26T01:52:36.522 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-26T01:52:36.522 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-26T01:52:36.522 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-26T01:52:36.522 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-26T01:52:36.522 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T01:52:36.522 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-26T01:52:36.522 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 33/34 2024-03-26T01:52:36.549 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-26T01:52:36.639 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-26T01:52:36.677 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-26T01:52:36.702 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-26T01:52:36.733 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T01:52:36.750 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T01:52:36.942 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T01:52:36.997 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T01:52:37.318 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-26T01:52:37.387 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T01:52:37.478 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T01:52:37.557 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 8/34 2024-03-26T01:52:37.589 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 9/34 2024-03-26T01:52:37.643 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 9/34 2024-03-26T01:52:37.649 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T01:52:37.649 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:52:37.649 INFO:teuthology.orchestra.run.smithi072.stdout:Upgraded: 2024-03-26T01:52:37.649 INFO:teuthology.orchestra.run.smithi072.stdout: librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:37.649 INFO:teuthology.orchestra.run.smithi072.stdout: librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:37.650 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-26T01:52:37.650 INFO:teuthology.orchestra.run.smithi072.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-26T01:52:37.650 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:37.650 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:37.650 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:37.650 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:37.650 INFO:teuthology.orchestra.run.smithi072.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-26T01:52:37.650 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-26T01:52:37.650 INFO:teuthology.orchestra.run.smithi072.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-26T01:52:37.650 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:37.650 INFO:teuthology.orchestra.run.smithi072.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-26T01:52:37.650 INFO:teuthology.orchestra.run.smithi072.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-26T01:52:37.650 INFO:teuthology.orchestra.run.smithi072.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-26T01:52:37.650 INFO:teuthology.orchestra.run.smithi072.stdout: libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:37.651 INFO:teuthology.orchestra.run.smithi072.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-26T01:52:37.651 INFO:teuthology.orchestra.run.smithi072.stdout: librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:37.651 INFO:teuthology.orchestra.run.smithi072.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-26T01:52:37.651 INFO:teuthology.orchestra.run.smithi072.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-26T01:52:37.651 INFO:teuthology.orchestra.run.smithi072.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-26T01:52:37.651 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:37.651 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:37.651 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:37.651 INFO:teuthology.orchestra.run.smithi072.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-26T01:52:37.651 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:37.651 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:37.651 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:37.652 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-26T01:52:37.652 INFO:teuthology.orchestra.run.smithi072.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-26T01:52:37.652 INFO:teuthology.orchestra.run.smithi072.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-26T01:52:37.652 INFO:teuthology.orchestra.run.smithi072.stdout: re2-1:20211101-3.el9.x86_64 2024-03-26T01:52:37.652 INFO:teuthology.orchestra.run.smithi072.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-26T01:52:37.652 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:52:37.652 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:52:37.672 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-26T01:52:37.717 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9. 11/34 2024-03-26T01:52:37.783 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 12/34 2024-03-26T01:52:37.808 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 13/34 2024-03-26T01:52:37.880 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 13/34 2024-03-26T01:52:37.915 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-26T01:52:38.023 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-26T01:52:38.038 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-test 2024-03-26T01:52:38.058 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-26T01:52:38.118 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-26T01:52:38.148 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-26T01:52:38.177 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-26T01:52:38.210 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-26T01:52:38.524 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-26T01:52:38.548 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 22/34 2024-03-26T01:52:38.587 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:00:31 ago on Tue 26 Mar 2024 01:52:07 AM UTC. 2024-03-26T01:52:38.638 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 22/34 2024-03-26T01:52:38.669 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 23/34 2024-03-26T01:52:38.712 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-26T01:52:38.743 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:52:38.743 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:52:38.744 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-03-26T01:52:38.744 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:52:38.744 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-26T01:52:38.744 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-test x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 47 M 2024-03-26T01:52:38.744 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-03-26T01:52:38.744 INFO:teuthology.orchestra.run.smithi072.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-26T01:52:38.744 INFO:teuthology.orchestra.run.smithi072.stdout: libcephsqlite x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 163 k 2024-03-26T01:52:38.744 INFO:teuthology.orchestra.run.smithi072.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-26T01:52:38.744 INFO:teuthology.orchestra.run.smithi072.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-26T01:52:38.744 INFO:teuthology.orchestra.run.smithi072.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-26T01:52:38.744 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:52:38.745 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-26T01:52:38.745 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:52:38.745 INFO:teuthology.orchestra.run.smithi072.stdout:Install 6 Packages 2024-03-26T01:52:38.745 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:52:38.745 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 48 M 2024-03-26T01:52:38.745 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 205 M 2024-03-26T01:52:38.746 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-26T01:52:39.167 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-26T01:52:39.209 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 26/34 2024-03-26T01:52:39.263 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 26/34 2024-03-26T01:52:39.308 INFO:teuthology.orchestra.run.smithi072.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 585 kB/s | 217 kB 00:00 2024-03-26T01:52:39.332 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 27/34 2024-03-26T01:52:39.362 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x8 28/34 2024-03-26T01:52:39.502 INFO:teuthology.orchestra.run.smithi072.stdout:(2/6): libcephsqlite-19.0.0-1775.g5e70124b.el9. 289 kB/s | 163 kB 00:00 2024-03-26T01:52:39.536 INFO:teuthology.orchestra.run.smithi072.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.3 MB/s | 300 kB 00:00 2024-03-26T01:52:39.595 INFO:teuthology.orchestra.run.smithi072.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 678 kB/s | 63 kB 00:00 2024-03-26T01:52:39.928 INFO:teuthology.orchestra.run.smithi072.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 477 kB/s | 187 kB 00:00 2024-03-26T01:52:41.513 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T01:52:41.542 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T01:52:41.588 INFO:teuthology.orchestra.run.smithi072.stdout:(6/6): ceph-test-19.0.0-1775.g5e70124b.el9.x86_ 18 MB/s | 47 MB 00:02 2024-03-26T01:52:41.592 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:52:41.592 INFO:teuthology.orchestra.run.smithi072.stdout:Total 17 MB/s | 48 MB 00:02 2024-03-26T01:52:41.606 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-26T01:52:41.627 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-26T01:52:41.627 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-26T01:52:42.000 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-26T01:52:42.001 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-26T01:52:42.093 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T01:52:42.138 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 30/34 2024-03-26T01:52:42.221 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 30/34 2024-03-26T01:52:42.221 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-26T01:52:42.221 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-26T01:52:42.221 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:52:42.256 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T01:52:42.604 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-26T01:52:42.687 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-26T01:52:42.740 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-26T01:52:42.772 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T01:52:42.817 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-26T01:52:42.838 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 5/6 2024-03-26T01:52:47.306 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 5/6 2024-03-26T01:52:47.401 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 6/6 2024-03-26T01:52:51.246 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 6/6 2024-03-26T01:52:51.246 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 1/6 2024-03-26T01:52:51.246 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 2/6 2024-03-26T01:52:51.246 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T01:52:51.246 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-26T01:52:51.246 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-26T01:52:51.690 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T01:52:51.690 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /sys 2024-03-26T01:52:51.690 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /proc 2024-03-26T01:52:51.690 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /mnt 2024-03-26T01:52:51.690 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /var/tmp 2024-03-26T01:52:51.690 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /home 2024-03-26T01:52:51.690 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /root 2024-03-26T01:52:51.690 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /tmp 2024-03-26T01:52:51.690 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:52:51.693 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-26T01:52:51.693 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:52:51.693 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-26T01:52:51.693 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:51.693 INFO:teuthology.orchestra.run.smithi072.stdout: jq-1.6-16.el9.x86_64 2024-03-26T01:52:51.693 INFO:teuthology.orchestra.run.smithi072.stdout: libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:51.693 INFO:teuthology.orchestra.run.smithi072.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-26T01:52:51.693 INFO:teuthology.orchestra.run.smithi072.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-26T01:52:51.693 INFO:teuthology.orchestra.run.smithi072.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-26T01:52:51.693 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:52:51.693 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:52:51.967 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph 2024-03-26T01:52:52.532 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:00:45 ago on Tue 26 Mar 2024 01:52:07 AM UTC. 2024-03-26T01:52:52.615 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 32/34 2024-03-26T01:52:52.640 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 32/34 2024-03-26T01:52:52.640 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T01:52:52.640 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-26T01:52:52.641 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T01:52:52.641 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T01:52:52.641 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:52:52.642 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T01:52:52.688 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:52:52.690 INFO:teuthology.orchestra.run.smithi072.stdout:========================================================================================= 2024-03-26T01:52:52.691 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-03-26T01:52:52.691 INFO:teuthology.orchestra.run.smithi072.stdout:========================================================================================= 2024-03-26T01:52:52.691 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-26T01:52:52.691 INFO:teuthology.orchestra.run.smithi072.stdout: ceph x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 6.6 k 2024-03-26T01:52:52.691 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-03-26T01:52:52.691 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mds x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 2.3 M 2024-03-26T01:52:52.691 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 1.5 M 2024-03-26T01:52:52.691 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 248 k 2024-03-26T01:52:52.691 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mon x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 4.7 M 2024-03-26T01:52:52.691 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-osd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 17 M 2024-03-26T01:52:52.692 INFO:teuthology.orchestra.run.smithi072.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-26T01:52:52.692 INFO:teuthology.orchestra.run.smithi072.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-26T01:52:52.692 INFO:teuthology.orchestra.run.smithi072.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-26T01:52:52.692 INFO:teuthology.orchestra.run.smithi072.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-26T01:52:52.692 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-26T01:52:52.692 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-26T01:52:52.692 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-26T01:52:52.692 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-26T01:52:52.692 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-26T01:52:52.692 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-26T01:52:52.692 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-26T01:52:52.692 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-26T01:52:52.693 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-26T01:52:52.693 INFO:teuthology.orchestra.run.smithi072.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-26T01:52:52.693 INFO:teuthology.orchestra.run.smithi072.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-26T01:52:52.693 INFO:teuthology.orchestra.run.smithi072.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-26T01:52:52.693 INFO:teuthology.orchestra.run.smithi072.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-26T01:52:52.693 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-26T01:52:52.693 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-26T01:52:52.693 INFO:teuthology.orchestra.run.smithi072.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-26T01:52:52.693 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-26T01:52:52.693 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-26T01:52:52.693 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-26T01:52:52.693 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-26T01:52:52.693 INFO:teuthology.orchestra.run.smithi072.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-26T01:52:52.693 INFO:teuthology.orchestra.run.smithi072.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-26T01:52:52.693 INFO:teuthology.orchestra.run.smithi072.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-26T01:52:52.694 INFO:teuthology.orchestra.run.smithi072.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-26T01:52:52.694 INFO:teuthology.orchestra.run.smithi072.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-26T01:52:52.694 INFO:teuthology.orchestra.run.smithi072.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-26T01:52:52.694 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:52:52.694 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-26T01:52:52.694 INFO:teuthology.orchestra.run.smithi072.stdout:========================================================================================= 2024-03-26T01:52:52.694 INFO:teuthology.orchestra.run.smithi072.stdout:Install 36 Packages 2024-03-26T01:52:52.694 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:52:52.695 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 30 M 2024-03-26T01:52:52.695 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 107 M 2024-03-26T01:52:52.695 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-26T01:52:52.727 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T01:52:52.727 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T01:52:53.434 INFO:teuthology.orchestra.run.smithi072.stdout:(1/36): ceph-19.0.0-1775.g5e70124b.el9.x86_64.r 31 kB/s | 6.6 kB 00:00 2024-03-26T01:52:53.844 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T01:52:53.844 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 1/34 2024-03-26T01:52:53.845 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2/34 2024-03-26T01:52:53.845 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 3/34 2024-03-26T01:52:53.845 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 4/34 2024-03-26T01:52:53.845 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 5/34 2024-03-26T01:52:53.845 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 6/34 2024-03-26T01:52:53.845 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T01:52:53.845 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9. 8/34 2024-03-26T01:52:53.845 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x8 9/34 2024-03-26T01:52:53.845 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 10/34 2024-03-26T01:52:53.845 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 11/34 2024-03-26T01:52:53.845 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 12/34 2024-03-26T01:52:53.846 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 13/34 2024-03-26T01:52:53.846 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-26T01:52:53.846 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-26T01:52:53.846 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-26T01:52:53.846 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-26T01:52:53.846 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-26T01:52:53.846 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-26T01:52:53.846 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-26T01:52:53.846 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-26T01:52:53.846 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-26T01:52:53.846 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-26T01:52:53.847 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-26T01:52:53.847 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-26T01:52:53.847 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-26T01:52:53.847 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-26T01:52:53.847 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-26T01:52:53.847 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-26T01:52:53.847 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-26T01:52:53.847 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T01:52:53.847 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-26T01:52:53.847 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 33/34 2024-03-26T01:52:53.860 INFO:teuthology.orchestra.run.smithi072.stdout:(2/36): ceph-mgr-19.0.0-1775.g5e70124b.el9.x86_ 2.3 MB/s | 1.5 MB 00:00 2024-03-26T01:52:54.752 INFO:teuthology.orchestra.run.smithi072.stdout:(3/36): ceph-mds-19.0.0-1775.g5e70124b.el9.x86_ 1.5 MB/s | 2.3 MB 00:01 2024-03-26T01:52:54.786 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T01:52:54.786 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:52:54.786 INFO:teuthology.orchestra.run.smithi173.stdout:Upgraded: 2024-03-26T01:52:54.786 INFO:teuthology.orchestra.run.smithi173.stdout: librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:54.786 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:54.787 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-26T01:52:54.787 INFO:teuthology.orchestra.run.smithi173.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-26T01:52:54.787 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:54.787 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:54.787 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:54.787 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:54.787 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-26T01:52:54.787 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-26T01:52:54.787 INFO:teuthology.orchestra.run.smithi173.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-26T01:52:54.787 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:54.787 INFO:teuthology.orchestra.run.smithi173.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-26T01:52:54.787 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-26T01:52:54.787 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-26T01:52:54.787 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:54.788 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-26T01:52:54.788 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:54.788 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-26T01:52:54.788 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-26T01:52:54.788 INFO:teuthology.orchestra.run.smithi173.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-26T01:52:54.788 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:54.788 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:54.788 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:54.788 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-26T01:52:54.788 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:54.788 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:54.788 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:52:54.788 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-26T01:52:54.788 INFO:teuthology.orchestra.run.smithi173.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-26T01:52:54.789 INFO:teuthology.orchestra.run.smithi173.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-26T01:52:54.789 INFO:teuthology.orchestra.run.smithi173.stdout: re2-1:20211101-3.el9.x86_64 2024-03-26T01:52:54.789 INFO:teuthology.orchestra.run.smithi173.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-26T01:52:54.789 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:52:54.789 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:52:55.120 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-test 2024-03-26T01:52:55.153 INFO:teuthology.orchestra.run.smithi072.stdout:(4/36): ceph-osd-19.0.0-1775.g5e70124b.el9.x86_ 13 MB/s | 17 MB 00:01 2024-03-26T01:52:55.195 INFO:teuthology.orchestra.run.smithi072.stdout:(5/36): ceph-mgr-modules-core-19.0.0-1775.g5e70 562 kB/s | 248 kB 00:00 2024-03-26T01:52:55.462 INFO:teuthology.orchestra.run.smithi072.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 605 kB/s | 187 kB 00:00 2024-03-26T01:52:55.487 INFO:teuthology.orchestra.run.smithi072.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 493 kB/s | 144 kB 00:00 2024-03-26T01:52:55.521 INFO:teuthology.orchestra.run.smithi072.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 537 kB/s | 32 kB 00:00 2024-03-26T01:52:55.547 INFO:teuthology.orchestra.run.smithi072.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 800 kB/s | 47 kB 00:00 2024-03-26T01:52:55.639 INFO:teuthology.orchestra.run.smithi072.stdout:(10/36): ceph-mon-19.0.0-1775.g5e70124b.el9.x86 2.1 MB/s | 4.7 MB 00:02 2024-03-26T01:52:55.656 INFO:teuthology.orchestra.run.smithi072.stdout:(11/36): python3-toml-0.10.2-6.el9.noarch.rpm 287 kB/s | 39 kB 00:00 2024-03-26T01:52:55.677 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:30 ago on Tue 26 Mar 2024 01:52:25 AM UTC. 2024-03-26T01:52:55.690 INFO:teuthology.orchestra.run.smithi072.stdout:(12/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.6 MB/s | 241 kB 00:00 2024-03-26T01:52:55.774 INFO:teuthology.orchestra.run.smithi072.stdout:(13/36): python3-cryptography-36.0.1-4.el9.x86_ 8.5 MB/s | 1.1 MB 00:00 2024-03-26T01:52:55.799 INFO:teuthology.orchestra.run.smithi072.stdout:(14/36): python3-ply-3.11-14.el9.noarch.rpm 950 kB/s | 103 kB 00:00 2024-03-26T01:52:55.826 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:52:55.826 INFO:teuthology.orchestra.run.smithi072.stdout:(15/36): python3-pycparser-2.20-6.el9.noarch.rp 918 kB/s | 124 kB 00:00 2024-03-26T01:52:55.827 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:52:55.827 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-26T01:52:55.827 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:52:55.827 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-26T01:52:55.827 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 47 M 2024-03-26T01:52:55.827 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-26T01:52:55.827 INFO:teuthology.orchestra.run.smithi173.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-26T01:52:55.827 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 163 k 2024-03-26T01:52:55.827 INFO:teuthology.orchestra.run.smithi173.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-26T01:52:55.827 INFO:teuthology.orchestra.run.smithi173.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-26T01:52:55.827 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-26T01:52:55.828 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:52:55.828 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-26T01:52:55.828 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:52:55.828 INFO:teuthology.orchestra.run.smithi173.stdout:Install 6 Packages 2024-03-26T01:52:55.828 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:52:55.828 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 48 M 2024-03-26T01:52:55.828 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 205 M 2024-03-26T01:52:55.829 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-26T01:52:55.852 INFO:teuthology.orchestra.run.smithi072.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.4 MB/s | 187 kB 00:00 2024-03-26T01:52:55.969 INFO:teuthology.orchestra.run.smithi072.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 157 kB/s | 22 kB 00:00 2024-03-26T01:52:56.003 INFO:teuthology.orchestra.run.smithi072.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-26T01:52:56.062 INFO:teuthology.orchestra.run.smithi072.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-03-26T01:52:56.087 INFO:teuthology.orchestra.run.smithi072.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 369 kB/s | 43 kB 00:00 2024-03-26T01:52:56.121 INFO:teuthology.orchestra.run.smithi072.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 5.9 MB/s | 358 kB 00:00 2024-03-26T01:52:56.147 INFO:teuthology.orchestra.run.smithi072.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 181 kB/s | 11 kB 00:00 2024-03-26T01:52:56.172 INFO:teuthology.orchestra.run.smithi072.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 349 kB/s | 18 kB 00:00 2024-03-26T01:52:56.198 INFO:teuthology.orchestra.run.smithi072.stdout:(24/36): python3-requests-2.25.1-8.el9.noarch.r 337 kB/s | 125 kB 00:00 2024-03-26T01:52:56.223 INFO:teuthology.orchestra.run.smithi072.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 304 kB/s | 23 kB 00:00 2024-03-26T01:52:56.249 INFO:teuthology.orchestra.run.smithi072.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 255 kB/s | 19 kB 00:00 2024-03-26T01:52:56.274 INFO:teuthology.orchestra.run.smithi072.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 258 kB/s | 20 kB 00:00 2024-03-26T01:52:56.291 INFO:teuthology.orchestra.run.smithi072.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 681 kB/s | 46 kB 00:00 2024-03-26T01:52:56.317 INFO:teuthology.orchestra.run.smithi072.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.1 MB/s | 79 kB 00:00 2024-03-26T01:52:56.327 INFO:teuthology.orchestra.run.smithi072.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 460 kB/s | 16 kB 00:00 2024-03-26T01:52:56.361 INFO:teuthology.orchestra.run.smithi072.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 1.0 MB/s | 36 kB 00:00 2024-03-26T01:52:56.363 INFO:teuthology.orchestra.run.smithi173.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 811 kB/s | 217 kB 00:00 2024-03-26T01:52:56.387 INFO:teuthology.orchestra.run.smithi072.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-03-26T01:52:56.412 INFO:teuthology.orchestra.run.smithi072.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 4.4 MB/s | 230 kB 00:00 2024-03-26T01:52:56.438 INFO:teuthology.orchestra.run.smithi072.stdout:(34/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-03-26T01:52:56.463 INFO:teuthology.orchestra.run.smithi072.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 396 kB/s | 20 kB 00:00 2024-03-26T01:52:56.471 INFO:teuthology.orchestra.run.smithi173.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.7 MB/s | 300 kB 00:00 2024-03-26T01:52:56.497 INFO:teuthology.orchestra.run.smithi072.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-03-26T01:52:56.500 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:52:56.500 INFO:teuthology.orchestra.run.smithi072.stdout:Total 7.8 MB/s | 30 MB 00:03 2024-03-26T01:52:56.525 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-26T01:52:56.530 INFO:teuthology.orchestra.run.smithi173.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2024-03-26T01:52:56.568 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-26T01:52:56.568 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-26T01:52:56.722 INFO:teuthology.orchestra.run.smithi173.stdout:(4/6): libcephsqlite-19.0.0-1775.g5e70124b.el9. 261 kB/s | 163 kB 00:00 2024-03-26T01:52:56.889 INFO:teuthology.orchestra.run.smithi173.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 522 kB/s | 187 kB 00:00 2024-03-26T01:52:57.007 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-26T01:52:57.007 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-26T01:52:58.380 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-26T01:52:58.426 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-26T01:52:58.458 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-26T01:52:58.487 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-26T01:52:58.567 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-26T01:52:58.614 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-26T01:52:58.647 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-26T01:52:58.738 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-26T01:52:58.827 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-26T01:52:58.876 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-26T01:52:58.949 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-26T01:52:59.006 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-26T01:52:59.053 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-26T01:52:59.110 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-26T01:52:59.170 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-26T01:52:59.377 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-26T01:52:59.429 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-26T01:52:59.476 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-26T01:52:59.512 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-26T01:52:59.542 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-26T01:52:59.617 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-26T01:52:59.655 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T01:52:59.690 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-26T01:52:59.731 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-26T01:52:59.774 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-26T01:52:59.810 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-26T01:52:59.891 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-26T01:52:59.933 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-26T01:53:00.008 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-26T01:53:00.156 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-26T01:53:00.282 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T01:53:00.452 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9. 31/36 2024-03-26T01:53:00.497 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 32/36 2024-03-26T01:53:00.522 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 32/36 2024-03-26T01:53:00.522 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T01:53:00.523 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-26T01:53:00.523 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T01:53:00.523 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T01:53:00.523 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:01.976 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 33/36 2024-03-26T01:53:02.002 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 33/36 2024-03-26T01:53:02.002 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T01:53:02.002 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-26T01:53:02.002 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T01:53:02.002 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T01:53:02.003 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:02.443 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 34/36 2024-03-26T01:53:02.467 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 34/36 2024-03-26T01:53:02.467 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T01:53:02.467 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-26T01:53:02.467 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T01:53:02.467 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T01:53:02.467 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:02.686 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 35/36 2024-03-26T01:53:02.709 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 35/36 2024-03-26T01:53:02.710 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T01:53:02.710 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-26T01:53:02.710 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T01:53:02.710 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T01:53:02.710 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:02.769 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 36/36 2024-03-26T01:53:05.307 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 36/36 2024-03-26T01:53:05.308 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 1/36 2024-03-26T01:53:05.308 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 2/36 2024-03-26T01:53:05.308 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 3/36 2024-03-26T01:53:05.308 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 4/36 2024-03-26T01:53:05.308 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 5/36 2024-03-26T01:53:05.308 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9. 6/36 2024-03-26T01:53:05.308 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-26T01:53:05.308 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-26T01:53:05.308 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-26T01:53:05.308 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-26T01:53:05.308 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-26T01:53:05.308 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-26T01:53:05.308 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-26T01:53:05.309 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-26T01:53:05.309 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-26T01:53:05.309 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-26T01:53:05.309 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-26T01:53:05.309 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-26T01:53:05.309 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-26T01:53:05.309 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-26T01:53:05.309 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T01:53:05.309 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-26T01:53:05.309 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-26T01:53:05.309 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-26T01:53:05.309 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-26T01:53:05.309 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-26T01:53:05.309 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-26T01:53:05.310 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-26T01:53:05.310 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-26T01:53:05.310 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T01:53:05.310 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-26T01:53:05.310 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-26T01:53:05.310 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-26T01:53:05.310 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-26T01:53:05.310 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-26T01:53:06.376 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-26T01:53:06.376 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:06.376 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-26T01:53:06.376 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:53:06.376 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:53:06.376 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:53:06.376 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T01:53:06.377 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:53:06.377 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:53:06.377 INFO:teuthology.orchestra.run.smithi072.stdout: lua-5.4.4-4.el9.x86_64 2024-03-26T01:53:06.377 INFO:teuthology.orchestra.run.smithi072.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-26T01:53:06.377 INFO:teuthology.orchestra.run.smithi072.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-26T01:53:06.377 INFO:teuthology.orchestra.run.smithi072.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-26T01:53:06.377 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-26T01:53:06.377 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-26T01:53:06.377 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-26T01:53:06.377 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-26T01:53:06.377 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-26T01:53:06.377 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-26T01:53:06.377 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-26T01:53:06.377 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-26T01:53:06.377 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-26T01:53:06.377 INFO:teuthology.orchestra.run.smithi072.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-26T01:53:06.378 INFO:teuthology.orchestra.run.smithi072.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-26T01:53:06.378 INFO:teuthology.orchestra.run.smithi072.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-26T01:53:06.378 INFO:teuthology.orchestra.run.smithi072.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-26T01:53:06.378 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-26T01:53:06.378 INFO:teuthology.orchestra.run.smithi072.stdout: python3-ply-3.11-14.el9.noarch 2024-03-26T01:53:06.378 INFO:teuthology.orchestra.run.smithi072.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-26T01:53:06.378 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-26T01:53:06.378 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-26T01:53:06.378 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-26T01:53:06.378 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-26T01:53:06.378 INFO:teuthology.orchestra.run.smithi072.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-26T01:53:06.378 INFO:teuthology.orchestra.run.smithi072.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-26T01:53:06.378 INFO:teuthology.orchestra.run.smithi072.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-26T01:53:06.379 INFO:teuthology.orchestra.run.smithi072.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-26T01:53:06.379 INFO:teuthology.orchestra.run.smithi072.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-26T01:53:06.379 INFO:teuthology.orchestra.run.smithi072.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-26T01:53:06.379 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:06.379 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:53:06.782 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-base 2024-03-26T01:53:07.345 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:00 ago on Tue 26 Mar 2024 01:52:07 AM UTC. 2024-03-26T01:53:07.442 INFO:teuthology.orchestra.run.smithi072.stdout:Package ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T01:53:07.495 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:53:07.496 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-03-26T01:53:07.496 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:53:07.578 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install cephadm 2024-03-26T01:53:08.131 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:01 ago on Tue 26 Mar 2024 01:52:07 AM UTC. 2024-03-26T01:53:08.278 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:53:08.279 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:53:08.279 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-03-26T01:53:08.279 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:53:08.279 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-26T01:53:08.279 INFO:teuthology.orchestra.run.smithi072.stdout: cephadm noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 748 k 2024-03-26T01:53:08.279 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:08.279 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-26T01:53:08.279 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:53:08.279 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2024-03-26T01:53:08.280 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:08.280 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 748 k 2024-03-26T01:53:08.280 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 754 k 2024-03-26T01:53:08.280 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-26T01:53:08.699 INFO:teuthology.orchestra.run.smithi072.stdout:cephadm-19.0.0-1775.g5e70124b.el9.noarch.rpm 1.7 MB/s | 748 kB 00:00 2024-03-26T01:53:08.699 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:53:08.699 INFO:teuthology.orchestra.run.smithi072.stdout:Total 1.7 MB/s | 748 kB 00:00 2024-03-26T01:53:08.699 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-26T01:53:08.704 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-26T01:53:08.705 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-26T01:53:08.710 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-26T01:53:08.711 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-26T01:53:08.787 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-26T01:53:09.029 INFO:teuthology.orchestra.run.smithi173.stdout:(6/6): ceph-test-19.0.0-1775.g5e70124b.el9.x86_ 3.7 MB/s | 47 MB 00:12 2024-03-26T01:53:09.032 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:53:09.032 INFO:teuthology.orchestra.run.smithi173.stdout:Total 3.7 MB/s | 48 MB 00:13 2024-03-26T01:53:09.046 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-26T01:53:09.068 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-26T01:53:09.068 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-26T01:53:09.192 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T01:53:09.284 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T01:53:09.447 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-26T01:53:09.447 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-26T01:53:09.742 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T01:53:10.028 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-26T01:53:10.041 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T01:53:10.041 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:10.041 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-26T01:53:10.041 INFO:teuthology.orchestra.run.smithi072.stdout: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T01:53:10.041 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:10.041 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:53:10.111 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-26T01:53:10.151 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-26T01:53:10.186 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T01:53:10.238 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-26T01:53:10.266 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 5/6 2024-03-26T01:53:10.348 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-immutable-object-cache 2024-03-26T01:53:10.917 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:03 ago on Tue 26 Mar 2024 01:52:07 AM UTC. 2024-03-26T01:53:11.068 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:53:11.069 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:53:11.069 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repo Size 2024-03-26T01:53:11.069 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:53:11.069 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-26T01:53:11.070 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 145 k 2024-03-26T01:53:11.070 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:11.070 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-26T01:53:11.070 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:53:11.070 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2024-03-26T01:53:11.070 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:11.070 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 145 k 2024-03-26T01:53:11.070 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 436 k 2024-03-26T01:53:11.070 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-26T01:53:11.415 INFO:teuthology.orchestra.run.smithi072.stdout:ceph-immutable-object-cache-19.0.0-1775.g5e7012 421 kB/s | 145 kB 00:00 2024-03-26T01:53:11.416 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:53:11.416 INFO:teuthology.orchestra.run.smithi072.stdout:Total 419 kB/s | 145 kB 00:00 2024-03-26T01:53:11.416 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-26T01:53:11.424 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-26T01:53:11.424 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-26T01:53:11.476 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-26T01:53:11.477 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-26T01:53:11.618 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-26T01:53:11.683 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T01:53:11.708 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T01:53:11.708 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T01:53:11.709 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-26T01:53:11.709 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:12.715 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T01:53:12.715 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:12.715 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-26T01:53:12.715 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:53:12.715 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:12.715 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:53:12.979 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-mgr 2024-03-26T01:53:13.543 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:06 ago on Tue 26 Mar 2024 01:52:07 AM UTC. 2024-03-26T01:53:13.641 INFO:teuthology.orchestra.run.smithi072.stdout:Package ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T01:53:13.694 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:53:13.695 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-03-26T01:53:13.695 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:53:13.774 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-mgr-dashboard 2024-03-26T01:53:14.327 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:07 ago on Tue 26 Mar 2024 01:52:07 AM UTC. 2024-03-26T01:53:14.477 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:53:14.478 INFO:teuthology.orchestra.run.smithi072.stdout:====================================================================================== 2024-03-26T01:53:14.478 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-03-26T01:53:14.479 INFO:teuthology.orchestra.run.smithi072.stdout:====================================================================================== 2024-03-26T01:53:14.479 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-26T01:53:14.479 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 1.8 M 2024-03-26T01:53:14.479 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-03-26T01:53:14.479 INFO:teuthology.orchestra.run.smithi072.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-26T01:53:14.479 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 31 k 2024-03-26T01:53:14.479 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 15 k 2024-03-26T01:53:14.479 INFO:teuthology.orchestra.run.smithi072.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-26T01:53:14.479 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-26T01:53:14.479 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-26T01:53:14.479 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-26T01:53:14.479 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-26T01:53:14.480 INFO:teuthology.orchestra.run.smithi072.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-26T01:53:14.480 INFO:teuthology.orchestra.run.smithi072.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-26T01:53:14.480 INFO:teuthology.orchestra.run.smithi072.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-26T01:53:14.480 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:14.480 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-26T01:53:14.480 INFO:teuthology.orchestra.run.smithi072.stdout:====================================================================================== 2024-03-26T01:53:14.480 INFO:teuthology.orchestra.run.smithi072.stdout:Install 12 Packages 2024-03-26T01:53:14.480 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:14.481 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 6.9 M 2024-03-26T01:53:14.481 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 26 M 2024-03-26T01:53:14.481 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-26T01:53:14.714 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 5/6 2024-03-26T01:53:14.790 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 6/6 2024-03-26T01:53:15.102 INFO:teuthology.orchestra.run.smithi072.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1775.g5e 160 kB/s | 31 kB 00:00 2024-03-26T01:53:15.202 INFO:teuthology.orchestra.run.smithi072.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1775.g5e7 52 kB/s | 15 kB 00:00 2024-03-26T01:53:15.270 INFO:teuthology.orchestra.run.smithi072.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.0 MB/s | 1.0 MB 00:00 2024-03-26T01:53:15.295 INFO:teuthology.orchestra.run.smithi072.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.5 MB/s | 238 kB 00:00 2024-03-26T01:53:15.471 INFO:teuthology.orchestra.run.smithi072.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.2 MB/s | 863 kB 00:00 2024-03-26T01:53:15.538 INFO:teuthology.orchestra.run.smithi072.stdout:(6/12): ceph-mgr-dashboard-19.0.0-1775.g5e70124 2.8 MB/s | 1.8 MB 00:00 2024-03-26T01:53:15.664 INFO:teuthology.orchestra.run.smithi072.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 101 kB/s | 19 kB 00:00 2024-03-26T01:53:15.757 INFO:teuthology.orchestra.run.smithi072.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.2 MB/s | 548 kB 00:00 2024-03-26T01:53:15.816 INFO:teuthology.orchestra.run.smithi072.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 525 kB/s | 31 kB 00:00 2024-03-26T01:53:15.842 INFO:teuthology.orchestra.run.smithi072.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 813 kB/s | 144 kB 00:00 2024-03-26T01:53:15.908 INFO:teuthology.orchestra.run.smithi072.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-03-26T01:53:16.009 INFO:teuthology.orchestra.run.smithi072.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.3 MB/s | 2.0 MB 00:00 2024-03-26T01:53:16.012 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:53:16.012 INFO:teuthology.orchestra.run.smithi072.stdout:Total 4.5 MB/s | 6.9 MB 00:01 2024-03-26T01:53:16.030 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-26T01:53:16.046 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-26T01:53:16.046 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-26T01:53:16.228 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-26T01:53:16.228 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-26T01:53:16.739 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-26T01:53:16.819 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-26T01:53:16.871 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-26T01:53:16.926 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-26T01:53:17.008 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-26T01:53:17.139 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-26T01:53:17.202 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 6/6 2024-03-26T01:53:17.202 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 1/6 2024-03-26T01:53:17.202 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 2/6 2024-03-26T01:53:17.202 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T01:53:17.202 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-26T01:53:17.202 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-26T01:53:17.278 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-26T01:53:17.376 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-26T01:53:17.429 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-26T01:53:17.458 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-26T01:53:17.490 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9 10/12 2024-03-26T01:53:17.595 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-26T01:53:17.595 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:17.595 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-26T01:53:17.595 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:53:17.595 INFO:teuthology.orchestra.run.smithi173.stdout: jq-1.6-16.el9.x86_64 2024-03-26T01:53:17.595 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:53:17.595 INFO:teuthology.orchestra.run.smithi173.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-26T01:53:17.595 INFO:teuthology.orchestra.run.smithi173.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-26T01:53:17.595 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-26T01:53:17.595 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:17.595 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:53:17.739 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el 11/12 2024-03-26T01:53:17.766 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 12/12 2024-03-26T01:53:17.858 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph 2024-03-26T01:53:18.426 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:53 ago on Tue 26 Mar 2024 01:52:25 AM UTC. 2024-03-26T01:53:18.578 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:53:18.581 INFO:teuthology.orchestra.run.smithi173.stdout:========================================================================================= 2024-03-26T01:53:18.581 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-26T01:53:18.581 INFO:teuthology.orchestra.run.smithi173.stdout:========================================================================================= 2024-03-26T01:53:18.581 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-26T01:53:18.581 INFO:teuthology.orchestra.run.smithi173.stdout: ceph x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 6.6 k 2024-03-26T01:53:18.581 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-26T01:53:18.581 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 2.3 M 2024-03-26T01:53:18.581 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 1.5 M 2024-03-26T01:53:18.581 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 248 k 2024-03-26T01:53:18.581 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 4.7 M 2024-03-26T01:53:18.582 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 17 M 2024-03-26T01:53:18.582 INFO:teuthology.orchestra.run.smithi173.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-26T01:53:18.582 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-26T01:53:18.582 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-26T01:53:18.582 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-26T01:53:18.582 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-26T01:53:18.582 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-26T01:53:18.582 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-26T01:53:18.582 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-26T01:53:18.582 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-26T01:53:18.582 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-26T01:53:18.582 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-26T01:53:18.582 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-26T01:53:18.583 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-26T01:53:18.583 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-26T01:53:18.583 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-26T01:53:18.583 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-26T01:53:18.583 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-26T01:53:18.583 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-26T01:53:18.583 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-26T01:53:18.583 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-26T01:53:18.583 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-26T01:53:18.583 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-26T01:53:18.583 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-26T01:53:18.583 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-26T01:53:18.583 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-26T01:53:18.583 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-26T01:53:18.583 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-26T01:53:18.584 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-26T01:53:18.584 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-26T01:53:18.584 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-26T01:53:18.584 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:18.584 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-26T01:53:18.584 INFO:teuthology.orchestra.run.smithi173.stdout:========================================================================================= 2024-03-26T01:53:18.584 INFO:teuthology.orchestra.run.smithi173.stdout:Install 36 Packages 2024-03-26T01:53:18.584 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:18.585 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 30 M 2024-03-26T01:53:18.585 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 107 M 2024-03-26T01:53:18.585 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-26T01:53:18.832 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 12/12 2024-03-26T01:53:18.832 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el 1/12 2024-03-26T01:53:18.832 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 2/12 2024-03-26T01:53:18.832 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9 3/12 2024-03-26T01:53:18.832 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-26T01:53:18.832 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-26T01:53:18.832 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-26T01:53:18.832 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-26T01:53:18.832 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-26T01:53:18.832 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-26T01:53:18.832 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-26T01:53:18.832 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-26T01:53:19.057 INFO:teuthology.orchestra.run.smithi173.stdout:(1/36): ceph-19.0.0-1775.g5e70124b.el9.x86_64.r 38 kB/s | 6.6 kB 00:00 2024-03-26T01:53:19.184 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-26T01:53:19.184 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:19.184 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-26T01:53:19.184 INFO:teuthology.orchestra.run.smithi072.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-26T01:53:19.184 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T01:53:19.184 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T01:53:19.184 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T01:53:19.185 INFO:teuthology.orchestra.run.smithi072.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-26T01:53:19.185 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-26T01:53:19.185 INFO:teuthology.orchestra.run.smithi072.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-26T01:53:19.185 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-26T01:53:19.185 INFO:teuthology.orchestra.run.smithi072.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-26T01:53:19.185 INFO:teuthology.orchestra.run.smithi072.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-26T01:53:19.185 INFO:teuthology.orchestra.run.smithi072.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-26T01:53:19.185 INFO:teuthology.orchestra.run.smithi072.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-26T01:53:19.185 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:19.185 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:53:19.316 INFO:teuthology.orchestra.run.smithi173.stdout:(2/36): ceph-mds-19.0.0-1775.g5e70124b.el9.x86_ 5.3 MB/s | 2.3 MB 00:00 2024-03-26T01:53:19.491 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-26T01:53:19.625 INFO:teuthology.orchestra.run.smithi173.stdout:(3/36): ceph-mon-19.0.0-1775.g5e70124b.el9.x86_ 8.3 MB/s | 4.7 MB 00:00 2024-03-26T01:53:19.683 INFO:teuthology.orchestra.run.smithi173.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1775.g5e70 4.2 MB/s | 248 kB 00:00 2024-03-26T01:53:19.800 INFO:teuthology.orchestra.run.smithi173.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-26T01:53:19.842 INFO:teuthology.orchestra.run.smithi173.stdout:(6/36): ceph-mgr-19.0.0-1775.g5e70124b.el9.x86_ 1.6 MB/s | 1.5 MB 00:00 2024-03-26T01:53:19.867 INFO:teuthology.orchestra.run.smithi173.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-03-26T01:53:19.893 INFO:teuthology.orchestra.run.smithi173.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 631 kB/s | 32 kB 00:00 2024-03-26T01:53:19.918 INFO:teuthology.orchestra.run.smithi173.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 942 kB/s | 47 kB 00:00 2024-03-26T01:53:19.943 INFO:teuthology.orchestra.run.smithi173.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 773 kB/s | 39 kB 00:00 2024-03-26T01:53:19.968 INFO:teuthology.orchestra.run.smithi173.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-26T01:53:20.027 INFO:teuthology.orchestra.run.smithi173.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-26T01:53:20.046 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:13 ago on Tue 26 Mar 2024 01:52:07 AM UTC. 2024-03-26T01:53:20.052 INFO:teuthology.orchestra.run.smithi173.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-26T01:53:20.078 INFO:teuthology.orchestra.run.smithi173.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-26T01:53:20.103 INFO:teuthology.orchestra.run.smithi173.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-26T01:53:20.196 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:53:20.197 INFO:teuthology.orchestra.run.smithi072.stdout:============================================================================================ 2024-03-26T01:53:20.197 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-03-26T01:53:20.197 INFO:teuthology.orchestra.run.smithi072.stdout:============================================================================================ 2024-03-26T01:53:20.197 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-26T01:53:20.197 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 7.4 M 2024-03-26T01:53:20.198 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-03-26T01:53:20.198 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-26T01:53:20.198 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-26T01:53:20.198 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-26T01:53:20.198 INFO:teuthology.orchestra.run.smithi072.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-26T01:53:20.198 INFO:teuthology.orchestra.run.smithi072.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-26T01:53:20.198 INFO:teuthology.orchestra.run.smithi072.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-26T01:53:20.198 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-26T01:53:20.198 INFO:teuthology.orchestra.run.smithi072.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-26T01:53:20.198 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-26T01:53:20.198 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-26T01:53:20.198 INFO:teuthology.orchestra.run.smithi072.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-26T01:53:20.198 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:20.199 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-26T01:53:20.199 INFO:teuthology.orchestra.run.smithi072.stdout:============================================================================================ 2024-03-26T01:53:20.199 INFO:teuthology.orchestra.run.smithi072.stdout:Install 12 Packages 2024-03-26T01:53:20.199 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:20.199 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 37 M 2024-03-26T01:53:20.199 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 209 M 2024-03-26T01:53:20.199 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-26T01:53:20.203 INFO:teuthology.orchestra.run.smithi173.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-03-26T01:53:20.228 INFO:teuthology.orchestra.run.smithi173.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 832 kB/s | 125 kB 00:00 2024-03-26T01:53:20.387 INFO:teuthology.orchestra.run.smithi173.stdout:(18/36): ceph-osd-19.0.0-1775.g5e70124b.el9.x86 16 MB/s | 17 MB 00:01 2024-03-26T01:53:20.498 INFO:teuthology.orchestra.run.smithi173.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 163 kB/s | 43 kB 00:00 2024-03-26T01:53:20.571 INFO:teuthology.orchestra.run.smithi173.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 411 kB/s | 151 kB 00:00 2024-03-26T01:53:20.646 INFO:teuthology.orchestra.run.smithi173.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 143 kB/s | 11 kB 00:00 2024-03-26T01:53:20.705 INFO:teuthology.orchestra.run.smithi173.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 541 kB/s | 172 kB 00:00 2024-03-26T01:53:20.738 INFO:teuthology.orchestra.run.smithi173.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 192 kB/s | 18 kB 00:00 2024-03-26T01:53:20.772 INFO:teuthology.orchestra.run.smithi072.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 132 kB/s | 32 kB 00:00 2024-03-26T01:53:20.805 INFO:teuthology.orchestra.run.smithi173.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-03-26T01:53:20.847 INFO:teuthology.orchestra.run.smithi173.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 179 kB/s | 19 kB 00:00 2024-03-26T01:53:20.864 INFO:teuthology.orchestra.run.smithi072.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 187 kB/s | 17 kB 00:00 2024-03-26T01:53:20.873 INFO:teuthology.orchestra.run.smithi173.stdout:(26/36): python3-cherrypy-18.6.1-2.el9.noarch.r 951 kB/s | 358 kB 00:00 2024-03-26T01:53:20.898 INFO:teuthology.orchestra.run.smithi173.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 213 kB/s | 20 kB 00:00 2024-03-26T01:53:20.923 INFO:teuthology.orchestra.run.smithi173.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 613 kB/s | 46 kB 00:00 2024-03-26T01:53:20.948 INFO:teuthology.orchestra.run.smithi072.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 408 kB/s | 34 kB 00:00 2024-03-26T01:53:20.998 INFO:teuthology.orchestra.run.smithi173.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 628 kB/s | 79 kB 00:00 2024-03-26T01:53:21.024 INFO:teuthology.orchestra.run.smithi173.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-03-26T01:53:21.116 INFO:teuthology.orchestra.run.smithi173.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 390 kB/s | 36 kB 00:00 2024-03-26T01:53:21.141 INFO:teuthology.orchestra.run.smithi173.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 633 kB/s | 90 kB 00:00 2024-03-26T01:53:21.182 INFO:teuthology.orchestra.run.smithi072.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.6 MB/s | 3.0 MB 00:00 2024-03-26T01:53:21.183 INFO:teuthology.orchestra.run.smithi173.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 957 kB/s | 272 kB 00:00 2024-03-26T01:53:21.300 INFO:teuthology.orchestra.run.smithi173.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.2 MB/s | 230 kB 00:00 2024-03-26T01:53:21.317 INFO:teuthology.orchestra.run.smithi173.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 150 kB/s | 20 kB 00:00 2024-03-26T01:53:21.574 INFO:teuthology.orchestra.run.smithi072.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 7.6 MB/s | 4.7 MB 00:00 2024-03-26T01:53:21.675 INFO:teuthology.orchestra.run.smithi072.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.7 MB/s | 4.8 MB 00:00 2024-03-26T01:53:21.708 INFO:teuthology.orchestra.run.smithi072.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.6 MB/s | 217 kB 00:00 2024-03-26T01:53:21.867 INFO:teuthology.orchestra.run.smithi173.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 589 kB/s | 427 kB 00:00 2024-03-26T01:53:21.869 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:53:21.869 INFO:teuthology.orchestra.run.smithi173.stdout:Total 9.1 MB/s | 30 MB 00:03 2024-03-26T01:53:21.894 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-26T01:53:21.936 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-26T01:53:21.936 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-26T01:53:22.126 INFO:teuthology.orchestra.run.smithi072.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-03-26T01:53:22.351 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-26T01:53:22.351 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-26T01:53:22.443 INFO:teuthology.orchestra.run.smithi072.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 617 kB/s | 195 kB 00:00 2024-03-26T01:53:22.577 INFO:teuthology.orchestra.run.smithi072.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.8 MB/s | 245 kB 00:00 2024-03-26T01:53:22.785 INFO:teuthology.orchestra.run.smithi072.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 747 kB/s | 803 kB 00:01 2024-03-26T01:53:23.722 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-26T01:53:23.786 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-26T01:53:23.818 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-26T01:53:23.847 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-26T01:53:23.908 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-26T01:53:23.954 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-26T01:53:23.986 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-26T01:53:24.078 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-26T01:53:24.164 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-26T01:53:24.206 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-26T01:53:24.269 INFO:teuthology.orchestra.run.smithi072.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 2.0 MB/s | 7.4 MB 00:03 2024-03-26T01:53:24.273 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:53:24.273 INFO:teuthology.orchestra.run.smithi072.stdout:Total 9.1 MB/s | 37 MB 00:04 2024-03-26T01:53:24.276 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-26T01:53:24.317 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-26T01:53:24.331 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-26T01:53:24.343 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-26T01:53:24.343 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-26T01:53:24.370 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-26T01:53:24.418 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-26T01:53:24.476 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-26T01:53:24.665 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-26T01:53:24.708 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-26T01:53:24.708 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-26T01:53:24.715 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-26T01:53:24.751 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-26T01:53:24.789 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-26T01:53:24.819 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-26T01:53:24.894 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-26T01:53:24.930 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T01:53:24.963 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-26T01:53:24.995 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-26T01:53:25.030 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-26T01:53:25.064 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-26T01:53:25.152 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-26T01:53:25.195 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-26T01:53:25.266 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-26T01:53:25.402 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-26T01:53:25.408 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-26T01:53:25.495 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-26T01:53:25.528 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T01:53:25.534 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-26T01:53:25.637 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-26T01:53:25.681 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-26T01:53:25.695 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9. 31/36 2024-03-26T01:53:25.717 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 32/36 2024-03-26T01:53:25.743 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 32/36 2024-03-26T01:53:25.743 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T01:53:25.743 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-26T01:53:25.743 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T01:53:25.743 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T01:53:25.743 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:25.851 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T01:53:25.881 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T01:53:25.967 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-26T01:53:26.635 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-26T01:53:26.752 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-26T01:53:27.193 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 33/36 2024-03-26T01:53:27.221 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 33/36 2024-03-26T01:53:27.221 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T01:53:27.221 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-26T01:53:27.221 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T01:53:27.221 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T01:53:27.221 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:27.678 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 34/36 2024-03-26T01:53:27.700 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 34/36 2024-03-26T01:53:27.700 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T01:53:27.700 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-26T01:53:27.700 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T01:53:27.700 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T01:53:27.701 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:27.919 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 35/36 2024-03-26T01:53:27.943 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 35/36 2024-03-26T01:53:27.943 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T01:53:27.943 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-26T01:53:27.943 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T01:53:27.943 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T01:53:27.943 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:27.991 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-26T01:53:28.008 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 36/36 2024-03-26T01:53:29.033 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-26T01:53:29.072 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 12/12 2024-03-26T01:53:29.797 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 36/36 2024-03-26T01:53:29.797 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 1/36 2024-03-26T01:53:29.797 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 2/36 2024-03-26T01:53:29.798 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 3/36 2024-03-26T01:53:29.798 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 4/36 2024-03-26T01:53:29.798 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 5/36 2024-03-26T01:53:29.798 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9. 6/36 2024-03-26T01:53:29.798 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-26T01:53:29.798 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-26T01:53:29.798 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-26T01:53:29.798 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-26T01:53:29.798 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-26T01:53:29.798 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-26T01:53:29.798 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-26T01:53:29.798 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-26T01:53:29.799 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-26T01:53:29.799 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-26T01:53:29.799 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-26T01:53:29.799 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-26T01:53:29.799 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-26T01:53:29.799 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-26T01:53:29.799 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T01:53:29.799 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-26T01:53:29.799 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-26T01:53:29.799 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-26T01:53:29.799 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-26T01:53:29.799 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-26T01:53:29.799 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-26T01:53:29.799 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-26T01:53:29.800 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-26T01:53:29.800 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T01:53:29.800 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-26T01:53:29.800 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-26T01:53:29.800 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-26T01:53:29.800 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-26T01:53:29.800 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-26T01:53:30.869 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-26T01:53:30.869 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:30.869 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-26T01:53:30.869 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:53:30.869 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:53:30.869 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:53:30.869 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T01:53:30.869 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:53:30.869 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:53:30.870 INFO:teuthology.orchestra.run.smithi173.stdout: lua-5.4.4-4.el9.x86_64 2024-03-26T01:53:30.870 INFO:teuthology.orchestra.run.smithi173.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-26T01:53:30.870 INFO:teuthology.orchestra.run.smithi173.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-26T01:53:30.870 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-26T01:53:30.870 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-26T01:53:30.870 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-26T01:53:30.870 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-26T01:53:30.870 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-26T01:53:30.870 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-26T01:53:30.870 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-26T01:53:30.870 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-26T01:53:30.870 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-26T01:53:30.870 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-26T01:53:30.871 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-26T01:53:30.871 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-26T01:53:30.871 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-26T01:53:30.871 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-26T01:53:30.871 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-26T01:53:30.871 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply-3.11-14.el9.noarch 2024-03-26T01:53:30.871 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-26T01:53:30.871 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-26T01:53:30.871 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-26T01:53:30.871 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-26T01:53:30.871 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-26T01:53:30.871 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-26T01:53:30.871 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-26T01:53:30.872 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-26T01:53:30.872 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-26T01:53:30.872 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-26T01:53:30.872 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-26T01:53:30.872 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:30.872 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:53:31.246 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-base 2024-03-26T01:53:31.811 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:06 ago on Tue 26 Mar 2024 01:52:25 AM UTC. 2024-03-26T01:53:31.908 INFO:teuthology.orchestra.run.smithi173.stdout:Package ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T01:53:31.960 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:53:31.961 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-26T01:53:31.961 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:53:32.024 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install cephadm 2024-03-26T01:53:32.584 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:07 ago on Tue 26 Mar 2024 01:52:25 AM UTC. 2024-03-26T01:53:32.733 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:53:32.734 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:53:32.734 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-26T01:53:32.734 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:53:32.734 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-26T01:53:32.734 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 748 k 2024-03-26T01:53:32.734 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:32.734 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-26T01:53:32.734 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:53:32.734 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-26T01:53:32.735 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:32.735 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 748 k 2024-03-26T01:53:32.735 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 754 k 2024-03-26T01:53:32.735 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-26T01:53:33.066 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 12/12 2024-03-26T01:53:33.066 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 1/12 2024-03-26T01:53:33.067 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-26T01:53:33.067 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-26T01:53:33.067 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-26T01:53:33.067 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T01:53:33.067 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T01:53:33.067 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-26T01:53:33.067 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-26T01:53:33.067 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-26T01:53:33.067 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-26T01:53:33.067 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-26T01:53:33.268 INFO:teuthology.orchestra.run.smithi173.stdout:cephadm-19.0.0-1775.g5e70124b.el9.noarch.rpm 1.4 MB/s | 748 kB 00:00 2024-03-26T01:53:33.268 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:53:33.269 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.4 MB/s | 748 kB 00:00 2024-03-26T01:53:33.269 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-26T01:53:33.274 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-26T01:53:33.274 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-26T01:53:33.280 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-26T01:53:33.280 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-26T01:53:33.351 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-26T01:53:33.552 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-26T01:53:33.552 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:33.552 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-26T01:53:33.552 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T01:53:33.552 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-26T01:53:33.552 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-26T01:53:33.552 INFO:teuthology.orchestra.run.smithi072.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-26T01:53:33.552 INFO:teuthology.orchestra.run.smithi072.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-26T01:53:33.552 INFO:teuthology.orchestra.run.smithi072.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-26T01:53:33.552 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-26T01:53:33.553 INFO:teuthology.orchestra.run.smithi072.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-26T01:53:33.553 INFO:teuthology.orchestra.run.smithi072.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-26T01:53:33.553 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-26T01:53:33.553 INFO:teuthology.orchestra.run.smithi072.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-26T01:53:33.553 INFO:teuthology.orchestra.run.smithi072.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-26T01:53:33.553 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:33.553 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:53:33.725 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T01:53:33.813 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T01:53:34.023 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-mgr-rook 2024-03-26T01:53:34.211 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T01:53:34.486 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T01:53:34.486 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:34.487 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-26T01:53:34.487 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T01:53:34.487 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:34.487 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:53:34.593 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:27 ago on Tue 26 Mar 2024 01:52:07 AM UTC. 2024-03-26T01:53:34.683 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-immutable-object-cache 2024-03-26T01:53:34.742 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:53:34.743 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================== 2024-03-26T01:53:34.743 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-03-26T01:53:34.744 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================== 2024-03-26T01:53:34.744 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-26T01:53:34.744 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-rook noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 49 k 2024-03-26T01:53:34.744 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-03-26T01:53:34.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-26T01:53:34.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-26T01:53:34.744 INFO:teuthology.orchestra.run.smithi072.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-26T01:53:34.745 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-26T01:53:34.745 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-26T01:53:34.745 INFO:teuthology.orchestra.run.smithi072.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-26T01:53:34.745 INFO:teuthology.orchestra.run.smithi072.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-26T01:53:34.745 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-26T01:53:34.745 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-26T01:53:34.745 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-26T01:53:34.745 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-26T01:53:34.745 INFO:teuthology.orchestra.run.smithi072.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-26T01:53:34.745 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:34.745 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-26T01:53:34.745 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================== 2024-03-26T01:53:34.745 INFO:teuthology.orchestra.run.smithi072.stdout:Install 13 Packages 2024-03-26T01:53:34.745 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:34.746 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 2.0 M 2024-03-26T01:53:34.746 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 26 M 2024-03-26T01:53:34.746 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-26T01:53:34.970 INFO:teuthology.orchestra.run.smithi072.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 158 kB/s | 24 kB 00:00 2024-03-26T01:53:34.995 INFO:teuthology.orchestra.run.smithi072.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 101 kB/s | 18 kB 00:00 2024-03-26T01:53:35.087 INFO:teuthology.orchestra.run.smithi072.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-03-26T01:53:35.113 INFO:teuthology.orchestra.run.smithi072.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 174 kB 00:00 2024-03-26T01:53:35.147 INFO:teuthology.orchestra.run.smithi072.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-26T01:53:35.172 INFO:teuthology.orchestra.run.smithi072.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 723 kB/s | 43 kB 00:00 2024-03-26T01:53:35.206 INFO:teuthology.orchestra.run.smithi072.stdout:(7/13): ceph-mgr-rook-19.0.0-1775.g5e70124b.el9 126 kB/s | 49 kB 00:00 2024-03-26T01:53:35.254 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:10 ago on Tue 26 Mar 2024 01:52:25 AM UTC. 2024-03-26T01:53:35.402 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:53:35.403 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:53:35.403 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-03-26T01:53:35.403 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:53:35.403 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-26T01:53:35.403 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 145 k 2024-03-26T01:53:35.403 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:35.404 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-26T01:53:35.404 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:53:35.404 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-26T01:53:35.404 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:35.404 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 145 k 2024-03-26T01:53:35.404 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 436 k 2024-03-26T01:53:35.404 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-26T01:53:35.490 INFO:teuthology.orchestra.run.smithi072.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 94 kB/s | 32 kB 00:00 2024-03-26T01:53:35.515 INFO:teuthology.orchestra.run.smithi072.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 44 kB/s | 15 kB 00:00 2024-03-26T01:53:35.691 INFO:teuthology.orchestra.run.smithi072.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 337 kB/s | 59 kB 00:00 2024-03-26T01:53:35.742 INFO:teuthology.orchestra.run.smithi072.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 379 kB/s | 203 kB 00:00 2024-03-26T01:53:35.842 INFO:teuthology.orchestra.run.smithi072.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 595 kB/s | 90 kB 00:00 2024-03-26T01:53:35.849 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-immutable-object-cache-19.0.0-1775.g5e7012 326 kB/s | 145 kB 00:00 2024-03-26T01:53:35.849 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:53:35.850 INFO:teuthology.orchestra.run.smithi173.stdout:Total 325 kB/s | 145 kB 00:00 2024-03-26T01:53:35.850 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-26T01:53:35.857 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-26T01:53:35.858 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-26T01:53:35.909 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-26T01:53:35.910 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-26T01:53:35.950 INFO:teuthology.orchestra.run.smithi072.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.2 MB/s | 1.0 MB 00:00 2024-03-26T01:53:35.955 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:53:35.955 INFO:teuthology.orchestra.run.smithi072.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-03-26T01:53:35.955 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-26T01:53:35.968 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-26T01:53:35.969 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-26T01:53:36.049 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-26T01:53:36.059 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-26T01:53:36.060 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-26T01:53:36.117 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T01:53:36.141 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T01:53:36.141 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T01:53:36.141 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-26T01:53:36.141 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:36.570 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-26T01:53:36.732 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-26T01:53:36.788 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-26T01:53:36.844 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-26T01:53:36.874 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-26T01:53:36.916 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-26T01:53:37.010 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-26T01:53:37.121 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T01:53:37.121 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:37.122 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-26T01:53:37.122 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:53:37.122 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:37.122 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:53:37.130 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-26T01:53:37.180 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-26T01:53:37.366 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr 2024-03-26T01:53:37.900 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-26T01:53:37.934 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:12 ago on Tue 26 Mar 2024 01:52:25 AM UTC. 2024-03-26T01:53:37.957 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-26T01:53:38.000 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-26T01:53:38.036 INFO:teuthology.orchestra.run.smithi173.stdout:Package ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T01:53:38.048 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-26T01:53:38.079 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 13/13 2024-03-26T01:53:38.089 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:53:38.091 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-26T01:53:38.091 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:53:38.149 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-dashboard 2024-03-26T01:53:38.711 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:13 ago on Tue 26 Mar 2024 01:52:25 AM UTC. 2024-03-26T01:53:38.876 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:53:38.877 INFO:teuthology.orchestra.run.smithi173.stdout:====================================================================================== 2024-03-26T01:53:38.877 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-26T01:53:38.877 INFO:teuthology.orchestra.run.smithi173.stdout:====================================================================================== 2024-03-26T01:53:38.877 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-26T01:53:38.877 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 1.8 M 2024-03-26T01:53:38.877 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-26T01:53:38.877 INFO:teuthology.orchestra.run.smithi173.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-26T01:53:38.877 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 31 k 2024-03-26T01:53:38.877 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 15 k 2024-03-26T01:53:38.877 INFO:teuthology.orchestra.run.smithi173.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-26T01:53:38.877 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-26T01:53:38.877 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-26T01:53:38.877 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-26T01:53:38.878 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-26T01:53:38.878 INFO:teuthology.orchestra.run.smithi173.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-26T01:53:38.878 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-26T01:53:38.878 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-26T01:53:38.878 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:38.878 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-26T01:53:38.878 INFO:teuthology.orchestra.run.smithi173.stdout:====================================================================================== 2024-03-26T01:53:38.878 INFO:teuthology.orchestra.run.smithi173.stdout:Install 12 Packages 2024-03-26T01:53:38.878 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:38.879 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 6.9 M 2024-03-26T01:53:38.879 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 26 M 2024-03-26T01:53:38.879 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-26T01:53:39.130 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 13/13 2024-03-26T01:53:39.130 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 1/13 2024-03-26T01:53:39.130 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-26T01:53:39.130 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-26T01:53:39.130 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-26T01:53:39.130 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-26T01:53:39.130 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-26T01:53:39.130 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-26T01:53:39.131 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-26T01:53:39.131 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-26T01:53:39.131 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-26T01:53:39.131 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-26T01:53:39.131 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-26T01:53:39.395 INFO:teuthology.orchestra.run.smithi173.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1775.g5e7 79 kB/s | 15 kB 00:00 2024-03-26T01:53:39.453 INFO:teuthology.orchestra.run.smithi173.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1775.g5e 124 kB/s | 31 kB 00:00 2024-03-26T01:53:39.487 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-26T01:53:39.488 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:39.488 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-26T01:53:39.488 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T01:53:39.488 INFO:teuthology.orchestra.run.smithi072.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-26T01:53:39.488 INFO:teuthology.orchestra.run.smithi072.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-26T01:53:39.488 INFO:teuthology.orchestra.run.smithi072.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-26T01:53:39.488 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-26T01:53:39.488 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-26T01:53:39.488 INFO:teuthology.orchestra.run.smithi072.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-26T01:53:39.488 INFO:teuthology.orchestra.run.smithi072.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-26T01:53:39.488 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-26T01:53:39.488 INFO:teuthology.orchestra.run.smithi072.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-26T01:53:39.489 INFO:teuthology.orchestra.run.smithi072.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-26T01:53:39.489 INFO:teuthology.orchestra.run.smithi072.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-26T01:53:39.489 INFO:teuthology.orchestra.run.smithi072.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-26T01:53:39.489 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:39.489 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:53:39.537 INFO:teuthology.orchestra.run.smithi173.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-26T01:53:39.595 INFO:teuthology.orchestra.run.smithi173.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-26T01:53:39.788 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-mgr-cephadm 2024-03-26T01:53:39.812 INFO:teuthology.orchestra.run.smithi173.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1775.g5e70124 2.9 MB/s | 1.8 MB 00:00 2024-03-26T01:53:39.862 INFO:teuthology.orchestra.run.smithi173.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.2 MB/s | 548 kB 00:00 2024-03-26T01:53:39.879 INFO:teuthology.orchestra.run.smithi173.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 292 kB/s | 19 kB 00:00 2024-03-26T01:53:39.963 INFO:teuthology.orchestra.run.smithi173.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 20 MB/s | 2.0 MB 00:00 2024-03-26T01:53:39.988 INFO:teuthology.orchestra.run.smithi173.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.3 MB/s | 144 kB 00:00 2024-03-26T01:53:40.005 INFO:teuthology.orchestra.run.smithi173.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 740 kB/s | 31 kB 00:00 2024-03-26T01:53:40.039 INFO:teuthology.orchestra.run.smithi173.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-03-26T01:53:40.156 INFO:teuthology.orchestra.run.smithi173.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.4 MB/s | 863 kB 00:00 2024-03-26T01:53:40.160 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:53:40.160 INFO:teuthology.orchestra.run.smithi173.stdout:Total 5.3 MB/s | 6.9 MB 00:01 2024-03-26T01:53:40.178 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-26T01:53:40.195 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-26T01:53:40.195 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-26T01:53:40.345 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:33 ago on Tue 26 Mar 2024 01:52:07 AM UTC. 2024-03-26T01:53:40.370 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-26T01:53:40.371 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-26T01:53:40.495 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:53:40.496 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================== 2024-03-26T01:53:40.496 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-03-26T01:53:40.496 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================== 2024-03-26T01:53:40.496 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-26T01:53:40.496 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 138 k 2024-03-26T01:53:40.496 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-03-26T01:53:40.496 INFO:teuthology.orchestra.run.smithi072.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-26T01:53:40.496 INFO:teuthology.orchestra.run.smithi072.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-26T01:53:40.497 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-26T01:53:40.497 INFO:teuthology.orchestra.run.smithi072.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-26T01:53:40.497 INFO:teuthology.orchestra.run.smithi072.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-26T01:53:40.497 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:40.497 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-26T01:53:40.497 INFO:teuthology.orchestra.run.smithi072.stdout:======================================================================================== 2024-03-26T01:53:40.497 INFO:teuthology.orchestra.run.smithi072.stdout:Install 6 Packages 2024-03-26T01:53:40.497 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:40.498 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 6.8 M 2024-03-26T01:53:40.498 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 33 M 2024-03-26T01:53:40.498 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-26T01:53:40.890 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-26T01:53:40.956 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-26T01:53:40.992 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-26T01:53:41.038 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-26T01:53:41.077 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-26T01:53:41.181 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-26T01:53:41.219 INFO:teuthology.orchestra.run.smithi072.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-26T01:53:41.278 INFO:teuthology.orchestra.run.smithi072.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 684 kB/s | 248 kB 00:00 2024-03-26T01:53:41.310 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-26T01:53:41.345 INFO:teuthology.orchestra.run.smithi072.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1775.g5e70124b.e 321 kB/s | 138 kB 00:00 2024-03-26T01:53:41.396 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-26T01:53:41.437 INFO:teuthology.orchestra.run.smithi072.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 362 kB/s | 58 kB 00:00 2024-03-26T01:53:41.448 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-26T01:53:41.485 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-26T01:53:41.496 INFO:teuthology.orchestra.run.smithi072.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 338 kB/s | 51 kB 00:00 2024-03-26T01:53:41.517 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9 10/12 2024-03-26T01:53:41.759 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el 11/12 2024-03-26T01:53:41.792 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 12/12 2024-03-26T01:53:42.488 INFO:teuthology.orchestra.run.smithi072.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 432 kB/s | 548 kB 00:01 2024-03-26T01:53:42.489 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:53:42.490 INFO:teuthology.orchestra.run.smithi072.stdout:Total 3.4 MB/s | 6.8 MB 00:01 2024-03-26T01:53:42.504 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-26T01:53:42.516 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-26T01:53:42.516 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-26T01:53:42.538 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 12/12 2024-03-26T01:53:42.538 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el 1/12 2024-03-26T01:53:42.539 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 2/12 2024-03-26T01:53:42.539 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9 3/12 2024-03-26T01:53:42.539 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-26T01:53:42.539 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-26T01:53:42.539 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-26T01:53:42.540 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-26T01:53:42.540 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-26T01:53:42.540 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-26T01:53:42.540 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-26T01:53:42.540 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-26T01:53:42.607 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-26T01:53:42.607 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-26T01:53:42.875 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-26T01:53:42.875 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:42.875 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-26T01:53:42.876 INFO:teuthology.orchestra.run.smithi173.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-26T01:53:42.876 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T01:53:42.876 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T01:53:42.876 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T01:53:42.876 INFO:teuthology.orchestra.run.smithi173.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-26T01:53:42.876 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-26T01:53:42.876 INFO:teuthology.orchestra.run.smithi173.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-26T01:53:42.876 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-26T01:53:42.876 INFO:teuthology.orchestra.run.smithi173.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-26T01:53:42.876 INFO:teuthology.orchestra.run.smithi173.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-26T01:53:42.876 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-26T01:53:42.876 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-26T01:53:42.877 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:42.877 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:53:42.906 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-26T01:53:43.084 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-26T01:53:43.133 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-26T01:53:43.141 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-26T01:53:43.654 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-26T01:53:43.701 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:18 ago on Tue 26 Mar 2024 01:52:25 AM UTC. 2024-03-26T01:53:43.732 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-26T01:53:43.802 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-26T01:53:43.834 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 6/6 2024-03-26T01:53:43.852 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:53:43.853 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-03-26T01:53:43.853 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-26T01:53:43.853 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-03-26T01:53:43.853 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-26T01:53:43.853 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 7.4 M 2024-03-26T01:53:43.853 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-26T01:53:43.854 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-26T01:53:43.854 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-26T01:53:43.854 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-26T01:53:43.854 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-26T01:53:43.854 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-26T01:53:43.854 INFO:teuthology.orchestra.run.smithi173.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-26T01:53:43.854 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-26T01:53:43.854 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-26T01:53:43.854 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-26T01:53:43.854 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-26T01:53:43.854 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-26T01:53:43.854 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:43.854 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-26T01:53:43.854 INFO:teuthology.orchestra.run.smithi173.stdout:============================================================================================ 2024-03-26T01:53:43.854 INFO:teuthology.orchestra.run.smithi173.stdout:Install 12 Packages 2024-03-26T01:53:43.855 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:43.855 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 37 M 2024-03-26T01:53:43.855 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 209 M 2024-03-26T01:53:43.855 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-26T01:53:44.195 INFO:teuthology.orchestra.run.smithi173.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 213 kB/s | 32 kB 00:00 2024-03-26T01:53:44.237 INFO:teuthology.orchestra.run.smithi173.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 422 kB/s | 17 kB 00:00 2024-03-26T01:53:44.296 INFO:teuthology.orchestra.run.smithi173.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-26T01:53:44.321 INFO:teuthology.orchestra.run.smithi173.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 422 kB/s | 34 kB 00:00 2024-03-26T01:53:44.505 INFO:teuthology.orchestra.run.smithi173.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2024-03-26T01:53:44.572 INFO:teuthology.orchestra.run.smithi173.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-26T01:53:44.597 INFO:teuthology.orchestra.run.smithi173.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-26T01:53:44.838 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 6/6 2024-03-26T01:53:44.838 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/6 2024-03-26T01:53:44.839 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-26T01:53:44.839 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-26T01:53:44.839 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-26T01:53:44.839 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-26T01:53:44.948 INFO:teuthology.orchestra.run.smithi173.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-26T01:53:45.072 INFO:teuthology.orchestra.run.smithi173.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-03-26T01:53:45.120 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-26T01:53:45.120 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:45.120 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-26T01:53:45.121 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T01:53:45.121 INFO:teuthology.orchestra.run.smithi072.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-26T01:53:45.121 INFO:teuthology.orchestra.run.smithi072.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-26T01:53:45.121 INFO:teuthology.orchestra.run.smithi072.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-26T01:53:45.121 INFO:teuthology.orchestra.run.smithi072.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-26T01:53:45.121 INFO:teuthology.orchestra.run.smithi072.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-26T01:53:45.121 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:45.121 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:53:45.173 INFO:teuthology.orchestra.run.smithi173.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 868 kB/s | 195 kB 00:00 2024-03-26T01:53:45.214 INFO:teuthology.orchestra.run.smithi173.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-03-26T01:53:45.435 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-fuse 2024-03-26T01:53:45.931 INFO:teuthology.orchestra.run.smithi173.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 3.9 MB/s | 7.4 MB 00:01 2024-03-26T01:53:45.934 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:53:45.934 INFO:teuthology.orchestra.run.smithi173.stdout:Total 18 MB/s | 37 MB 00:02 2024-03-26T01:53:45.975 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-26T01:53:45.990 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:38 ago on Tue 26 Mar 2024 01:52:07 AM UTC. 2024-03-26T01:53:45.999 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-26T01:53:46.000 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-26T01:53:46.139 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:53:46.140 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:53:46.140 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-03-26T01:53:46.140 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:53:46.140 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-26T01:53:46.140 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-fuse x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 888 k 2024-03-26T01:53:46.140 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-03-26T01:53:46.140 INFO:teuthology.orchestra.run.smithi072.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-26T01:53:46.140 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:46.141 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-26T01:53:46.141 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:53:46.141 INFO:teuthology.orchestra.run.smithi072.stdout:Install 2 Packages 2024-03-26T01:53:46.141 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:46.141 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 967 k 2024-03-26T01:53:46.141 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 2.8 M 2024-03-26T01:53:46.141 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-26T01:53:46.260 INFO:teuthology.orchestra.run.smithi072.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 662 kB/s | 79 kB 00:00 2024-03-26T01:53:46.363 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-26T01:53:46.363 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-26T01:53:46.627 INFO:teuthology.orchestra.run.smithi072.stdout:(2/2): ceph-fuse-19.0.0-1775.g5e70124b.el9.x86_ 1.8 MB/s | 888 kB 00:00 2024-03-26T01:53:46.628 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:53:46.629 INFO:teuthology.orchestra.run.smithi072.stdout:Total 1.9 MB/s | 967 kB 00:00 2024-03-26T01:53:46.629 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-26T01:53:46.649 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-26T01:53:46.649 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-26T01:53:46.721 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-26T01:53:46.722 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-26T01:53:46.901 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-26T01:53:47.058 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-26T01:53:47.062 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-26T01:53:47.129 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-26T01:53:47.130 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2/2 2024-03-26T01:53:47.161 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-26T01:53:47.254 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-26T01:53:47.289 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-26T01:53:47.467 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T01:53:47.496 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T01:53:47.573 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-26T01:53:47.860 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2/2 2024-03-26T01:53:47.860 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/2 2024-03-26T01:53:48.160 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-26T01:53:48.160 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:48.160 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-26T01:53:48.160 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-26T01:53:48.160 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:48.160 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:53:48.226 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-26T01:53:48.334 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-26T01:53:48.518 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install ceph-volume 2024-03-26T01:53:49.090 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:42 ago on Tue 26 Mar 2024 01:52:07 AM UTC. 2024-03-26T01:53:49.247 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:53:49.248 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:53:49.248 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repository Size 2024-03-26T01:53:49.248 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:53:49.248 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-26T01:53:49.248 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-volume noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 264 k 2024-03-26T01:53:49.248 INFO:teuthology.orchestra.run.smithi072.stdout:Installing dependencies: 2024-03-26T01:53:49.248 INFO:teuthology.orchestra.run.smithi072.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-26T01:53:49.248 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:49.248 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-26T01:53:49.248 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:53:49.248 INFO:teuthology.orchestra.run.smithi072.stdout:Install 2 Packages 2024-03-26T01:53:49.249 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:49.249 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 332 k 2024-03-26T01:53:49.249 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 1.5 M 2024-03-26T01:53:49.249 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-26T01:53:49.370 INFO:teuthology.orchestra.run.smithi072.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 577 kB/s | 69 kB 00:00 2024-03-26T01:53:49.502 INFO:teuthology.orchestra.run.smithi072.stdout:(2/2): ceph-volume-19.0.0-1775.g5e70124b.el9.no 1.0 MB/s | 264 kB 00:00 2024-03-26T01:53:49.504 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:53:49.504 INFO:teuthology.orchestra.run.smithi072.stdout:Total 1.3 MB/s | 332 kB 00:00 2024-03-26T01:53:49.505 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-26T01:53:49.513 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-26T01:53:49.513 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-26T01:53:49.555 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-26T01:53:49.556 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-26T01:53:49.560 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-26T01:53:49.708 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-26T01:53:49.898 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-26T01:53:49.941 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2/2 2024-03-26T01:53:49.957 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2/2 2024-03-26T01:53:49.957 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T01:53:49.957 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-26T01:53:49.958 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:50.594 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-26T01:53:50.620 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 12/12 2024-03-26T01:53:50.830 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 1/2 2024-03-26T01:53:51.163 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-26T01:53:51.163 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:51.163 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-26T01:53:51.163 INFO:teuthology.orchestra.run.smithi072.stdout: ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T01:53:51.163 INFO:teuthology.orchestra.run.smithi072.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-26T01:53:51.163 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:51.163 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:53:51.536 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install librados-devel 2024-03-26T01:53:52.114 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:45 ago on Tue 26 Mar 2024 01:52:07 AM UTC. 2024-03-26T01:53:52.262 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:53:52.263 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:53:52.263 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repo Size 2024-03-26T01:53:52.263 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:53:52.263 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-26T01:53:52.263 INFO:teuthology.orchestra.run.smithi072.stdout: librados-devel x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 127 k 2024-03-26T01:53:52.263 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:52.263 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-26T01:53:52.263 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:53:52.263 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2024-03-26T01:53:52.264 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:52.264 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 127 k 2024-03-26T01:53:52.264 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 456 k 2024-03-26T01:53:52.264 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-26T01:53:52.485 INFO:teuthology.orchestra.run.smithi072.stdout:librados-devel-19.0.0-1775.g5e70124b.el9.x86_64 573 kB/s | 127 kB 00:00 2024-03-26T01:53:52.486 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:53:52.486 INFO:teuthology.orchestra.run.smithi072.stdout:Total 569 kB/s | 127 kB 00:00 2024-03-26T01:53:52.486 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-26T01:53:52.492 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-26T01:53:52.492 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-26T01:53:52.549 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-26T01:53:52.549 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-26T01:53:52.754 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-26T01:53:52.852 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:53:53.242 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 12/12 2024-03-26T01:53:53.242 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 1/12 2024-03-26T01:53:53.242 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-26T01:53:53.242 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-26T01:53:53.242 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-26T01:53:53.242 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T01:53:53.242 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T01:53:53.242 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-26T01:53:53.242 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-26T01:53:53.242 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-26T01:53:53.242 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-26T01:53:53.243 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-26T01:53:53.340 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:53:53.646 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:53:53.646 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:53.646 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-26T01:53:53.646 INFO:teuthology.orchestra.run.smithi072.stdout: librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:53:53.646 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:53.647 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:53:53.703 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-26T01:53:53.703 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:53.703 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-26T01:53:53.703 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T01:53:53.703 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-26T01:53:53.703 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-26T01:53:53.703 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-26T01:53:53.703 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-26T01:53:53.703 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-26T01:53:53.703 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-26T01:53:53.703 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-26T01:53:53.703 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-26T01:53:53.703 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-26T01:53:53.704 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-26T01:53:53.704 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-26T01:53:53.704 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:53.704 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:53:53.901 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install libcephfs2 2024-03-26T01:53:54.093 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-rook 2024-03-26T01:53:54.464 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:47 ago on Tue 26 Mar 2024 01:52:07 AM UTC. 2024-03-26T01:53:54.561 INFO:teuthology.orchestra.run.smithi072.stdout:Package libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T01:53:54.614 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:53:54.615 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-03-26T01:53:54.615 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:53:54.668 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:29 ago on Tue 26 Mar 2024 01:52:25 AM UTC. 2024-03-26T01:53:54.685 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install libcephfs-devel 2024-03-26T01:53:54.819 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:53:54.820 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-03-26T01:53:54.821 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-26T01:53:54.821 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-03-26T01:53:54.821 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-26T01:53:54.821 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 49 k 2024-03-26T01:53:54.821 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-26T01:53:54.821 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-26T01:53:54.821 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-26T01:53:54.821 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-26T01:53:54.821 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-26T01:53:54.821 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-26T01:53:54.822 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-26T01:53:54.822 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-26T01:53:54.822 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-26T01:53:54.822 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-26T01:53:54.822 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-26T01:53:54.822 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-26T01:53:54.822 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-26T01:53:54.822 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:54.822 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-26T01:53:54.822 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-03-26T01:53:54.822 INFO:teuthology.orchestra.run.smithi173.stdout:Install 13 Packages 2024-03-26T01:53:54.822 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:54.823 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 2.0 M 2024-03-26T01:53:54.823 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 26 M 2024-03-26T01:53:54.823 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-26T01:53:54.980 INFO:teuthology.orchestra.run.smithi173.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 164 kB/s | 18 kB 00:00 2024-03-26T01:53:54.997 INFO:teuthology.orchestra.run.smithi173.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 192 kB/s | 24 kB 00:00 2024-03-26T01:53:55.047 INFO:teuthology.orchestra.run.smithi173.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-26T01:53:55.072 INFO:teuthology.orchestra.run.smithi173.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-26T01:53:55.098 INFO:teuthology.orchestra.run.smithi173.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-26T01:53:55.123 INFO:teuthology.orchestra.run.smithi173.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 849 kB/s | 43 kB 00:00 2024-03-26T01:53:55.149 INFO:teuthology.orchestra.run.smithi173.stdout:(7/13): ceph-mgr-rook-19.0.0-1775.g5e70124b.el9 177 kB/s | 49 kB 00:00 2024-03-26T01:53:55.199 INFO:teuthology.orchestra.run.smithi173.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 200 kB/s | 15 kB 00:00 2024-03-26T01:53:55.224 INFO:teuthology.orchestra.run.smithi173.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 255 kB/s | 32 kB 00:00 2024-03-26T01:53:55.243 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:48 ago on Tue 26 Mar 2024 01:52:07 AM UTC. 2024-03-26T01:53:55.291 INFO:teuthology.orchestra.run.smithi173.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.4 MB/s | 203 kB 00:00 2024-03-26T01:53:55.316 INFO:teuthology.orchestra.run.smithi173.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 642 kB/s | 59 kB 00:00 2024-03-26T01:53:55.341 INFO:teuthology.orchestra.run.smithi173.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-03-26T01:53:55.391 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:53:55.391 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:53:55.391 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repo Size 2024-03-26T01:53:55.391 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:53:55.391 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-26T01:53:55.391 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs-devel x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 31 k 2024-03-26T01:53:55.392 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:55.392 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-26T01:53:55.392 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:53:55.392 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2024-03-26T01:53:55.392 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:55.392 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 31 k 2024-03-26T01:53:55.392 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 140 k 2024-03-26T01:53:55.392 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-26T01:53:55.458 INFO:teuthology.orchestra.run.smithi173.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.9 MB/s | 1.0 MB 00:00 2024-03-26T01:53:55.460 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:53:55.461 INFO:teuthology.orchestra.run.smithi173.stdout:Total 3.2 MB/s | 2.0 MB 00:00 2024-03-26T01:53:55.461 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-26T01:53:55.473 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-26T01:53:55.474 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-26T01:53:55.563 INFO:teuthology.orchestra.run.smithi072.stdout:libcephfs-devel-19.0.0-1775.g5e70124b.el9.x86_6 183 kB/s | 31 kB 00:00 2024-03-26T01:53:55.563 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:53:55.564 INFO:teuthology.orchestra.run.smithi072.stdout:Total 181 kB/s | 31 kB 00:00 2024-03-26T01:53:55.564 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-26T01:53:55.565 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-26T01:53:55.565 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-26T01:53:55.567 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-26T01:53:55.568 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-26T01:53:55.596 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-26T01:53:55.597 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-26T01:53:55.695 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-26T01:53:55.821 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:53:56.084 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-26T01:53:56.238 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-26T01:53:56.249 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:53:56.301 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-26T01:53:56.357 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-26T01:53:56.394 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-26T01:53:56.435 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-26T01:53:56.501 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:53:56.501 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:56.501 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-26T01:53:56.501 INFO:teuthology.orchestra.run.smithi072.stdout: libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:53:56.501 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:53:56.502 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:53:56.528 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-26T01:53:56.637 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-26T01:53:56.686 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-26T01:53:56.698 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install librados2 2024-03-26T01:53:57.258 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:50 ago on Tue 26 Mar 2024 01:52:07 AM UTC. 2024-03-26T01:53:57.354 INFO:teuthology.orchestra.run.smithi072.stdout:Package librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T01:53:57.405 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:53:57.406 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-03-26T01:53:57.406 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:53:57.417 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-26T01:53:57.459 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-26T01:53:57.488 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install librbd1 2024-03-26T01:53:57.492 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-26T01:53:57.541 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-26T01:53:57.562 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 13/13 2024-03-26T01:53:58.049 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:51 ago on Tue 26 Mar 2024 01:52:07 AM UTC. 2024-03-26T01:53:58.145 INFO:teuthology.orchestra.run.smithi072.stdout:Package librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T01:53:58.196 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:53:58.197 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-03-26T01:53:58.197 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:53:58.279 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install python3-rados 2024-03-26T01:53:58.354 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 13/13 2024-03-26T01:53:58.354 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 1/13 2024-03-26T01:53:58.354 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-26T01:53:58.354 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-26T01:53:58.354 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-26T01:53:58.354 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-26T01:53:58.355 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-26T01:53:58.355 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-26T01:53:58.355 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-26T01:53:58.355 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-26T01:53:58.355 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-26T01:53:58.355 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-26T01:53:58.355 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-26T01:53:58.733 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-26T01:53:58.733 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:58.733 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-26T01:53:58.733 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T01:53:58.733 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-26T01:53:58.734 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-26T01:53:58.734 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-26T01:53:58.734 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-26T01:53:58.734 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-26T01:53:58.734 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-26T01:53:58.734 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-26T01:53:58.734 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-26T01:53:58.734 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-26T01:53:58.734 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-26T01:53:58.734 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-26T01:53:58.734 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-26T01:53:58.734 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:58.734 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:53:58.842 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:51 ago on Tue 26 Mar 2024 01:52:07 AM UTC. 2024-03-26T01:53:58.938 INFO:teuthology.orchestra.run.smithi072.stdout:Package python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T01:53:58.980 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-cephadm 2024-03-26T01:53:58.989 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:53:58.990 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-03-26T01:53:58.990 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:53:59.067 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install python3-rgw 2024-03-26T01:53:59.537 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:34 ago on Tue 26 Mar 2024 01:52:25 AM UTC. 2024-03-26T01:53:59.625 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:52 ago on Tue 26 Mar 2024 01:52:07 AM UTC. 2024-03-26T01:53:59.686 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:53:59.687 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-03-26T01:53:59.687 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-26T01:53:59.687 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-03-26T01:53:59.687 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-26T01:53:59.687 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 138 k 2024-03-26T01:53:59.687 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-26T01:53:59.688 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-26T01:53:59.688 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-26T01:53:59.688 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-26T01:53:59.688 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-26T01:53:59.688 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-26T01:53:59.688 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:59.688 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-26T01:53:59.688 INFO:teuthology.orchestra.run.smithi173.stdout:======================================================================================== 2024-03-26T01:53:59.688 INFO:teuthology.orchestra.run.smithi173.stdout:Install 6 Packages 2024-03-26T01:53:59.688 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:53:59.689 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 6.8 M 2024-03-26T01:53:59.689 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 33 M 2024-03-26T01:53:59.689 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-26T01:53:59.721 INFO:teuthology.orchestra.run.smithi072.stdout:Package python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T01:53:59.772 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:53:59.773 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-03-26T01:53:59.773 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:53:59.855 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install python3-cephfs 2024-03-26T01:54:00.168 INFO:teuthology.orchestra.run.smithi173.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-03-26T01:54:00.409 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:53 ago on Tue 26 Mar 2024 01:52:07 AM UTC. 2024-03-26T01:54:00.485 INFO:teuthology.orchestra.run.smithi173.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 5.8 MB 00:00 2024-03-26T01:54:00.508 INFO:teuthology.orchestra.run.smithi072.stdout:Package python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T01:54:00.510 INFO:teuthology.orchestra.run.smithi173.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1775.g5e70124b.e 271 kB/s | 138 kB 00:00 2024-03-26T01:54:00.559 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:54:00.560 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-03-26T01:54:00.560 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:54:00.611 INFO:teuthology.orchestra.run.smithi173.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.2 MB/s | 548 kB 00:00 2024-03-26T01:54:00.627 INFO:teuthology.orchestra.run.smithi173.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 406 kB/s | 58 kB 00:00 2024-03-26T01:54:00.637 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install python3-rbd 2024-03-26T01:54:00.644 INFO:teuthology.orchestra.run.smithi173.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 382 kB/s | 51 kB 00:00 2024-03-26T01:54:00.646 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:54:00.647 INFO:teuthology.orchestra.run.smithi173.stdout:Total 7.1 MB/s | 6.8 MB 00:00 2024-03-26T01:54:00.660 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-26T01:54:00.669 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-26T01:54:00.670 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-26T01:54:00.757 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-26T01:54:00.757 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-26T01:54:01.072 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-26T01:54:01.198 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:54 ago on Tue 26 Mar 2024 01:52:07 AM UTC. 2024-03-26T01:54:01.240 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-26T01:54:01.286 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-26T01:54:01.294 INFO:teuthology.orchestra.run.smithi072.stdout:Package python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T01:54:01.346 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:54:01.346 INFO:teuthology.orchestra.run.smithi072.stdout:Nothing to do. 2024-03-26T01:54:01.346 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:54:01.419 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install rbd-fuse 2024-03-26T01:54:01.800 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-26T01:54:01.889 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-26T01:54:01.953 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-26T01:54:01.976 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 6/6 2024-03-26T01:54:01.981 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:54 ago on Tue 26 Mar 2024 01:52:07 AM UTC. 2024-03-26T01:54:02.128 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:54:02.129 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:54:02.129 INFO:teuthology.orchestra.run.smithi072.stdout: Package Architecture Version Repository Size 2024-03-26T01:54:02.129 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:54:02.129 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-26T01:54:02.130 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-fuse x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 85 k 2024-03-26T01:54:02.130 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:54:02.130 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-26T01:54:02.130 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:54:02.130 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2024-03-26T01:54:02.130 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:54:02.130 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 85 k 2024-03-26T01:54:02.130 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 231 k 2024-03-26T01:54:02.130 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-26T01:54:02.622 INFO:teuthology.orchestra.run.smithi072.stdout:rbd-fuse-19.0.0-1775.g5e70124b.el9.x86_64.rpm 173 kB/s | 85 kB 00:00 2024-03-26T01:54:02.623 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:54:02.623 INFO:teuthology.orchestra.run.smithi072.stdout:Total 173 kB/s | 85 kB 00:00 2024-03-26T01:54:02.623 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-26T01:54:02.630 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-26T01:54:02.630 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-26T01:54:02.687 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-26T01:54:02.687 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-26T01:54:02.823 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-26T01:54:02.889 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 6/6 2024-03-26T01:54:02.889 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/6 2024-03-26T01:54:02.889 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-26T01:54:02.889 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-26T01:54:02.889 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-26T01:54:02.890 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-26T01:54:02.921 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:54:03.189 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-26T01:54:03.189 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:03.189 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-26T01:54:03.189 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T01:54:03.189 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-26T01:54:03.189 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-26T01:54:03.189 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-26T01:54:03.190 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-26T01:54:03.190 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-26T01:54:03.190 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:03.190 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:54:03.336 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:54:03.444 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-fuse 2024-03-26T01:54:03.642 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:54:03.642 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:54:03.642 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-26T01:54:03.642 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:54:03.642 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:54:03.642 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:54:03.953 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install rbd-mirror 2024-03-26T01:54:04.006 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:39 ago on Tue 26 Mar 2024 01:52:25 AM UTC. 2024-03-26T01:54:04.156 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:54:04.157 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:54:04.157 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-26T01:54:04.157 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:54:04.157 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-26T01:54:04.157 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 888 k 2024-03-26T01:54:04.157 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-26T01:54:04.157 INFO:teuthology.orchestra.run.smithi173.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-26T01:54:04.157 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:04.157 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-26T01:54:04.157 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:54:04.157 INFO:teuthology.orchestra.run.smithi173.stdout:Install 2 Packages 2024-03-26T01:54:04.157 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:04.157 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 967 k 2024-03-26T01:54:04.157 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 2.8 M 2024-03-26T01:54:04.158 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-26T01:54:04.439 INFO:teuthology.orchestra.run.smithi173.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 280 kB/s | 79 kB 00:00 2024-03-26T01:54:04.518 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:01:57 ago on Tue 26 Mar 2024 01:52:07 AM UTC. 2024-03-26T01:54:04.671 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:54:04.672 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:54:04.672 INFO:teuthology.orchestra.run.smithi072.stdout: Package Arch Version Repo Size 2024-03-26T01:54:04.672 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:54:04.672 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-26T01:54:04.672 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-mirror x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.1 M 2024-03-26T01:54:04.672 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:54:04.672 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-26T01:54:04.672 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:54:04.672 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2024-03-26T01:54:04.672 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:54:04.672 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 3.1 M 2024-03-26T01:54:04.672 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 13 M 2024-03-26T01:54:04.672 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-26T01:54:05.439 INFO:teuthology.orchestra.run.smithi173.stdout:(2/2): ceph-fuse-19.0.0-1775.g5e70124b.el9.x86_ 693 kB/s | 888 kB 00:01 2024-03-26T01:54:05.440 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:54:05.441 INFO:teuthology.orchestra.run.smithi173.stdout:Total 753 kB/s | 967 kB 00:01 2024-03-26T01:54:05.441 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-26T01:54:05.464 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-26T01:54:05.464 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-26T01:54:05.490 INFO:teuthology.orchestra.run.smithi072.stdout:rbd-mirror-19.0.0-1775.g5e70124b.el9.x86_64.rpm 3.8 MB/s | 3.1 MB 00:00 2024-03-26T01:54:05.490 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:54:05.491 INFO:teuthology.orchestra.run.smithi072.stdout:Total 3.8 MB/s | 3.1 MB 00:00 2024-03-26T01:54:05.491 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-26T01:54:05.501 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-26T01:54:05.501 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-26T01:54:05.540 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-26T01:54:05.541 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-26T01:54:05.576 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-26T01:54:05.576 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-26T01:54:05.719 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-26T01:54:05.874 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-26T01:54:05.938 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2/2 2024-03-26T01:54:05.999 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-26T01:54:06.058 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:54:06.082 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:54:06.082 INFO:teuthology.orchestra.run.smithi072.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T01:54:06.083 INFO:teuthology.orchestra.run.smithi072.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-26T01:54:06.083 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T01:54:06.083 INFO:teuthology.orchestra.run.smithi072.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T01:54:06.083 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:54:06.626 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2/2 2024-03-26T01:54:06.626 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/2 2024-03-26T01:54:06.922 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-26T01:54:06.922 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:06.922 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-26T01:54:06.922 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-26T01:54:06.922 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:06.922 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:54:07.201 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-volume 2024-03-26T01:54:07.350 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:54:07.350 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:54:07.351 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-26T01:54:07.351 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:54:07.351 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:54:07.351 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:54:07.618 DEBUG:teuthology.orchestra.run.smithi072:> sudo yum -y install rbd-nbd 2024-03-26T01:54:07.778 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:42 ago on Tue 26 Mar 2024 01:52:25 AM UTC. 2024-03-26T01:54:07.932 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:54:07.933 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:54:07.933 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-03-26T01:54:07.933 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:54:07.933 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-26T01:54:07.934 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 264 k 2024-03-26T01:54:07.934 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-03-26T01:54:07.934 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-26T01:54:07.934 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:07.934 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-26T01:54:07.934 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:54:07.934 INFO:teuthology.orchestra.run.smithi173.stdout:Install 2 Packages 2024-03-26T01:54:07.934 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:07.934 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 332 k 2024-03-26T01:54:07.934 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 1.5 M 2024-03-26T01:54:07.934 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-26T01:54:08.190 INFO:teuthology.orchestra.run.smithi072.stdout:Last metadata expiration check: 0:02:01 ago on Tue 26 Mar 2024 01:52:07 AM UTC. 2024-03-26T01:54:08.210 INFO:teuthology.orchestra.run.smithi173.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 249 kB/s | 69 kB 00:00 2024-03-26T01:54:08.235 INFO:teuthology.orchestra.run.smithi173.stdout:(2/2): ceph-volume-19.0.0-1775.g5e70124b.el9.no 876 kB/s | 264 kB 00:00 2024-03-26T01:54:08.236 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:54:08.237 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.1 MB/s | 332 kB 00:00 2024-03-26T01:54:08.237 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-26T01:54:08.245 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-26T01:54:08.245 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-26T01:54:08.285 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-26T01:54:08.285 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-26T01:54:08.338 INFO:teuthology.orchestra.run.smithi072.stdout:Dependencies resolved. 2024-03-26T01:54:08.339 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:54:08.339 INFO:teuthology.orchestra.run.smithi072.stdout: Package Architecture Version Repository Size 2024-03-26T01:54:08.339 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:54:08.339 INFO:teuthology.orchestra.run.smithi072.stdout:Installing: 2024-03-26T01:54:08.339 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-nbd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 171 k 2024-03-26T01:54:08.339 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:54:08.339 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction Summary 2024-03-26T01:54:08.339 INFO:teuthology.orchestra.run.smithi072.stdout:================================================================================ 2024-03-26T01:54:08.339 INFO:teuthology.orchestra.run.smithi072.stdout:Install 1 Package 2024-03-26T01:54:08.339 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:54:08.340 INFO:teuthology.orchestra.run.smithi072.stdout:Total download size: 171 k 2024-03-26T01:54:08.340 INFO:teuthology.orchestra.run.smithi072.stdout:Installed size: 494 k 2024-03-26T01:54:08.340 INFO:teuthology.orchestra.run.smithi072.stdout:Downloading Packages: 2024-03-26T01:54:08.431 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-26T01:54:08.599 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-26T01:54:08.630 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2/2 2024-03-26T01:54:08.645 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2/2 2024-03-26T01:54:08.645 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T01:54:08.646 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-26T01:54:08.646 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:08.783 INFO:teuthology.orchestra.run.smithi072.stdout:rbd-nbd-19.0.0-1775.g5e70124b.el9.x86_64.rpm 385 kB/s | 171 kB 00:00 2024-03-26T01:54:08.784 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:54:08.784 INFO:teuthology.orchestra.run.smithi072.stdout:Total 384 kB/s | 171 kB 00:00 2024-03-26T01:54:08.784 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction check 2024-03-26T01:54:08.792 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction check succeeded. 2024-03-26T01:54:08.792 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction test 2024-03-26T01:54:08.849 INFO:teuthology.orchestra.run.smithi072.stdout:Transaction test succeeded. 2024-03-26T01:54:08.849 INFO:teuthology.orchestra.run.smithi072.stdout:Running transaction 2024-03-26T01:54:08.993 INFO:teuthology.orchestra.run.smithi072.stdout: Preparing : 1/1 2024-03-26T01:54:09.093 INFO:teuthology.orchestra.run.smithi072.stdout: Installing : rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:54:09.407 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 1/2 2024-03-26T01:54:09.570 INFO:teuthology.orchestra.run.smithi072.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:54:09.741 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-26T01:54:09.742 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:09.742 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-26T01:54:09.742 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T01:54:09.742 INFO:teuthology.orchestra.run.smithi173.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-26T01:54:09.742 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:09.742 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:54:09.858 INFO:teuthology.orchestra.run.smithi072.stdout: Verifying : rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:54:09.858 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:54:09.858 INFO:teuthology.orchestra.run.smithi072.stdout:Installed: 2024-03-26T01:54:09.859 INFO:teuthology.orchestra.run.smithi072.stdout: rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:54:09.859 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:54:09.859 INFO:teuthology.orchestra.run.smithi072.stdout:Complete! 2024-03-26T01:54:10.100 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librados-devel 2024-03-26T01:54:10.153 DEBUG:teuthology.parallel:result is None 2024-03-26T01:54:10.677 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:45 ago on Tue 26 Mar 2024 01:52:25 AM UTC. 2024-03-26T01:54:10.825 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:54:10.825 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:54:10.825 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-03-26T01:54:10.826 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:54:10.826 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-26T01:54:10.826 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 127 k 2024-03-26T01:54:10.826 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:10.826 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-26T01:54:10.826 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:54:10.826 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-26T01:54:10.826 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:10.826 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 127 k 2024-03-26T01:54:10.826 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 456 k 2024-03-26T01:54:10.826 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-26T01:54:11.291 INFO:teuthology.orchestra.run.smithi173.stdout:librados-devel-19.0.0-1775.g5e70124b.el9.x86_64 273 kB/s | 127 kB 00:00 2024-03-26T01:54:11.292 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:54:11.292 INFO:teuthology.orchestra.run.smithi173.stdout:Total 272 kB/s | 127 kB 00:00 2024-03-26T01:54:11.292 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-26T01:54:11.298 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-26T01:54:11.298 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-26T01:54:11.353 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-26T01:54:11.353 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-26T01:54:11.532 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-26T01:54:11.641 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:54:12.135 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:54:12.414 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:54:12.414 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:12.414 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-26T01:54:12.414 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:54:12.414 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:12.414 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:54:12.652 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install libcephfs2 2024-03-26T01:54:13.231 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:48 ago on Tue 26 Mar 2024 01:52:25 AM UTC. 2024-03-26T01:54:13.332 INFO:teuthology.orchestra.run.smithi173.stdout:Package libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T01:54:13.385 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:54:13.386 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-26T01:54:13.386 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:54:13.449 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install libcephfs-devel 2024-03-26T01:54:14.008 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:49 ago on Tue 26 Mar 2024 01:52:25 AM UTC. 2024-03-26T01:54:14.158 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:54:14.158 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:54:14.159 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-03-26T01:54:14.159 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:54:14.159 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-26T01:54:14.159 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 31 k 2024-03-26T01:54:14.159 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:14.159 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-26T01:54:14.159 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:54:14.159 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-26T01:54:14.159 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:14.159 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 31 k 2024-03-26T01:54:14.159 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 140 k 2024-03-26T01:54:14.159 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-26T01:54:14.420 INFO:teuthology.orchestra.run.smithi173.stdout:libcephfs-devel-19.0.0-1775.g5e70124b.el9.x86_6 120 kB/s | 31 kB 00:00 2024-03-26T01:54:14.420 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:54:14.420 INFO:teuthology.orchestra.run.smithi173.stdout:Total 120 kB/s | 31 kB 00:00 2024-03-26T01:54:14.421 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-26T01:54:14.424 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-26T01:54:14.424 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-26T01:54:14.455 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-26T01:54:14.455 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-26T01:54:14.548 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-26T01:54:14.659 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:54:15.057 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:54:15.304 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:54:15.304 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:15.305 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-26T01:54:15.305 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:54:15.305 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:15.305 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:54:15.495 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librados2 2024-03-26T01:54:16.058 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:51 ago on Tue 26 Mar 2024 01:52:25 AM UTC. 2024-03-26T01:54:16.154 INFO:teuthology.orchestra.run.smithi173.stdout:Package librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T01:54:16.205 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:54:16.206 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-26T01:54:16.207 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:54:16.267 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librbd1 2024-03-26T01:54:16.828 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:51 ago on Tue 26 Mar 2024 01:52:25 AM UTC. 2024-03-26T01:54:16.924 INFO:teuthology.orchestra.run.smithi173.stdout:Package librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T01:54:16.975 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:54:16.976 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-26T01:54:16.977 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:54:17.033 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rados 2024-03-26T01:54:17.601 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:52 ago on Tue 26 Mar 2024 01:52:25 AM UTC. 2024-03-26T01:54:17.699 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T01:54:17.750 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:54:17.751 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-26T01:54:17.751 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:54:17.812 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rgw 2024-03-26T01:54:18.377 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:53 ago on Tue 26 Mar 2024 01:52:25 AM UTC. 2024-03-26T01:54:18.474 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T01:54:18.526 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:54:18.527 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-26T01:54:18.528 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:54:18.584 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-cephfs 2024-03-26T01:54:19.147 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:54 ago on Tue 26 Mar 2024 01:52:25 AM UTC. 2024-03-26T01:54:19.245 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T01:54:19.296 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:54:19.297 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-26T01:54:19.298 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:54:19.362 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rbd 2024-03-26T01:54:19.924 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:54 ago on Tue 26 Mar 2024 01:52:25 AM UTC. 2024-03-26T01:54:20.019 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T01:54:20.071 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:54:20.072 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-03-26T01:54:20.072 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:54:20.132 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-fuse 2024-03-26T01:54:20.704 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:55 ago on Tue 26 Mar 2024 01:52:25 AM UTC. 2024-03-26T01:54:20.861 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:54:20.862 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:54:20.862 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-03-26T01:54:20.862 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:54:20.862 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-26T01:54:20.862 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 85 k 2024-03-26T01:54:20.862 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:20.862 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-26T01:54:20.862 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:54:20.862 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-26T01:54:20.863 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:20.863 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 85 k 2024-03-26T01:54:20.863 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 231 k 2024-03-26T01:54:20.863 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-26T01:54:21.351 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-fuse-19.0.0-1775.g5e70124b.el9.x86_64.rpm 175 kB/s | 85 kB 00:00 2024-03-26T01:54:21.351 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:54:21.352 INFO:teuthology.orchestra.run.smithi173.stdout:Total 174 kB/s | 85 kB 00:00 2024-03-26T01:54:21.352 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-26T01:54:21.359 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-26T01:54:21.359 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-26T01:54:21.417 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-26T01:54:21.417 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-26T01:54:21.559 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-26T01:54:21.673 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:54:22.094 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:54:22.369 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:54:22.369 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:22.369 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-26T01:54:22.369 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:54:22.369 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:22.369 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:54:22.595 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-mirror 2024-03-26T01:54:23.167 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:58 ago on Tue 26 Mar 2024 01:52:25 AM UTC. 2024-03-26T01:54:23.320 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:54:23.321 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:54:23.321 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-03-26T01:54:23.321 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:54:23.321 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-26T01:54:23.321 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.1 M 2024-03-26T01:54:23.321 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:23.321 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-26T01:54:23.321 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:54:23.321 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-26T01:54:23.321 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:23.321 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 3.1 M 2024-03-26T01:54:23.321 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 13 M 2024-03-26T01:54:23.321 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-26T01:54:24.151 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-mirror-19.0.0-1775.g5e70124b.el9.x86_64.rpm 3.7 MB/s | 3.1 MB 00:00 2024-03-26T01:54:24.151 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:54:24.152 INFO:teuthology.orchestra.run.smithi173.stdout:Total 3.7 MB/s | 3.1 MB 00:00 2024-03-26T01:54:24.152 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-26T01:54:24.161 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-26T01:54:24.161 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-26T01:54:24.237 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-26T01:54:24.237 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-26T01:54:24.666 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-26T01:54:24.718 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:54:24.743 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:54:24.743 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T01:54:24.743 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-26T01:54:24.743 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T01:54:24.743 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T01:54:24.743 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:25.870 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:54:25.870 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:25.870 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-26T01:54:25.870 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:54:25.870 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:25.870 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:54:26.098 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-nbd 2024-03-26T01:54:26.675 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:02:01 ago on Tue 26 Mar 2024 01:52:25 AM UTC. 2024-03-26T01:54:26.823 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-03-26T01:54:26.824 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:54:26.824 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-03-26T01:54:26.824 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:54:26.824 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-03-26T01:54:26.824 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 171 k 2024-03-26T01:54:26.824 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:26.824 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-03-26T01:54:26.824 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-03-26T01:54:26.824 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-03-26T01:54:26.824 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:26.824 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 171 k 2024-03-26T01:54:26.824 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 494 k 2024-03-26T01:54:26.825 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-03-26T01:54:27.045 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-nbd-19.0.0-1775.g5e70124b.el9.x86_64.rpm 775 kB/s | 171 kB 00:00 2024-03-26T01:54:27.046 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:54:27.046 INFO:teuthology.orchestra.run.smithi173.stdout:Total 770 kB/s | 171 kB 00:00 2024-03-26T01:54:27.046 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-03-26T01:54:27.053 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-03-26T01:54:27.054 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-03-26T01:54:27.110 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-03-26T01:54:27.110 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-03-26T01:54:27.256 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-03-26T01:54:27.360 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:54:27.786 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:54:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T01:54:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:28.067 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-03-26T01:54:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T01:54:28.067 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:54:28.067 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-03-26T01:54:28.306 DEBUG:teuthology.parallel:result is None 2024-03-26T01:54:28.306 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T01:54:28.450 DEBUG:teuthology.orchestra.run.smithi072:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-26T01:54:28.475 INFO:teuthology.orchestra.run.smithi072.stdout:19.0.0-1775.g5e70124b.el9 2024-03-26T01:54:28.475 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1775.g5e70124b.el9 2024-03-26T01:54:28.475 INFO:teuthology.task.install:The correct ceph version 19.0.0-1775.g5e70124b is installed. 2024-03-26T01:54:28.477 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T01:54:28.692 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-26T01:54:28.712 INFO:teuthology.orchestra.run.smithi173.stdout:19.0.0-1775.g5e70124b.el9 2024-03-26T01:54:28.713 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1775.g5e70124b.el9 2024-03-26T01:54:28.713 INFO:teuthology.task.install:The correct ceph version 19.0.0-1775.g5e70124b is installed. 2024-03-26T01:54:28.715 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-26T01:54:28.715 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-26T01:54:28.715 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-26T01:54:28.747 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-26T01:54:28.748 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-26T01:54:28.780 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-26T01:54:28.781 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-26T01:54:28.781 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/daemon-helper 2024-03-26T01:54:28.817 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-26T01:54:28.887 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-26T01:54:28.887 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/daemon-helper 2024-03-26T01:54:28.914 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-26T01:54:28.982 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-26T01:54:28.982 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-26T01:54:28.982 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-26T01:54:29.012 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-26T01:54:29.082 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-26T01:54:29.082 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-26T01:54:29.108 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-26T01:54:29.177 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-26T01:54:29.177 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-26T01:54:29.177 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/usr/bin/stdin-killer 2024-03-26T01:54:29.204 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-26T01:54:29.272 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-26T01:54:29.272 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/stdin-killer 2024-03-26T01:54:29.298 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-26T01:54:29.366 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-26T01:54:29.473 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1}, 'global': {'mon election default strategy': 3, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a', 'cephadm_mode': 'cephadm-package'} 2024-03-26T01:54:29.473 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T01:54:29.473 INFO:tasks.cephadm:Cluster fsid is c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T01:54:29.473 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-26T01:54:29.474 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.72', 'mon.c': '[v2:172.21.15.72:3301,v1:172.21.15.72:6790]', 'mon.b': '172.21.15.173'} 2024-03-26T01:54:29.474 INFO:tasks.cephadm:First mon is mon.a on smithi072 2024-03-26T01:54:29.474 INFO:tasks.cephadm:First mgr is y 2024-03-26T01:54:29.474 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-26T01:54:29.474 DEBUG:teuthology.orchestra.run.smithi072:> sudo hostname $(hostname -s) 2024-03-26T01:54:29.503 DEBUG:teuthology.orchestra.run.smithi173:> sudo hostname $(hostname -s) 2024-03-26T01:54:29.541 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a on all hosts... 2024-03-26T01:54:29.541 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a pull 2024-03-26T01:54:29.545 DEBUG:teuthology.orchestra.run.smithi173:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a pull 2024-03-26T01:54:30.379 INFO:teuthology.orchestra.run.smithi072.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T01:54:30.548 INFO:teuthology.orchestra.run.smithi173.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T01:55:03.268 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-03-26T01:55:03.268 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T01:55:03.268 INFO:teuthology.orchestra.run.smithi173.stdout: "image_id": "5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767", 2024-03-26T01:55:03.268 INFO:teuthology.orchestra.run.smithi173.stdout: "repo_digests": [ 2024-03-26T01:55:03.269 INFO:teuthology.orchestra.run.smithi173.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2" 2024-03-26T01:55:03.269 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-03-26T01:55:03.269 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-03-26T01:55:06.037 INFO:teuthology.orchestra.run.smithi072.stdout:{ 2024-03-26T01:55:06.037 INFO:teuthology.orchestra.run.smithi072.stdout: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T01:55:06.037 INFO:teuthology.orchestra.run.smithi072.stdout: "image_id": "5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767", 2024-03-26T01:55:06.037 INFO:teuthology.orchestra.run.smithi072.stdout: "repo_digests": [ 2024-03-26T01:55:06.037 INFO:teuthology.orchestra.run.smithi072.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2" 2024-03-26T01:55:06.037 INFO:teuthology.orchestra.run.smithi072.stdout: ] 2024-03-26T01:55:06.037 INFO:teuthology.orchestra.run.smithi072.stdout:} 2024-03-26T01:55:06.062 DEBUG:teuthology.orchestra.run.smithi072:> sudo mkdir -p /etc/ceph 2024-03-26T01:55:06.106 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph 2024-03-26T01:55:06.141 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 777 /etc/ceph 2024-03-26T01:55:06.173 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /etc/ceph 2024-03-26T01:55:06.212 INFO:tasks.cephadm:Writing seed config... 2024-03-26T01:55:06.213 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-26T01:55:06.213 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-26T01:55:06.213 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-26T01:55:06.213 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-03-26T01:55:06.213 INFO:tasks.cephadm: override: [global] ms bind msgr1 = False 2024-03-26T01:55:06.213 INFO:tasks.cephadm: override: [global] ms bind msgr2 = True 2024-03-26T01:55:06.213 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-26T01:55:06.213 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-26T01:55:06.213 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-26T01:55:06.214 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-26T01:55:06.214 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-03-26T01:55:06.214 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-26T01:55:06.214 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-26T01:55:06.214 INFO:tasks.cephadm: override: [osd] osd class default list = * 2024-03-26T01:55:06.214 INFO:tasks.cephadm: override: [osd] osd class load list = * 2024-03-26T01:55:06.214 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-26T01:55:06.214 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-26T01:55:06.215 DEBUG:teuthology.orchestra.run.smithi072:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-26T01:55:06.229 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = c86fbdd4-eb13-11ee-95d0-87774f69a715 mon election default strategy = 3 ms bind msgr1 = False ms bind msgr2 = True ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd class default list = * osd class load list = * [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 2024-03-26T01:55:06.229 DEBUG:teuthology.orchestra.run.smithi072:mon.a> sudo journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@mon.a.service 2024-03-26T01:55:06.272 DEBUG:teuthology.orchestra.run.smithi072:mgr.y> sudo journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@mgr.y.service 2024-03-26T01:55:06.314 INFO:tasks.cephadm:Bootstrapping... 2024-03-26T01:55:06.315 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a -v bootstrap --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.72 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-26T01:55:06.507 INFO:teuthology.orchestra.run.smithi072.stdout:-------------------------------------------------------------------------------- 2024-03-26T01:55:06.507 INFO:teuthology.orchestra.run.smithi072.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a', '-v', 'bootstrap', '--fsid', 'c86fbdd4-eb13-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.72', '--skip-admin-label'] 2024-03-26T01:55:06.531 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T01:55:06.532 INFO:teuthology.orchestra.run.smithi072.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-26T01:55:06.532 INFO:teuthology.orchestra.run.smithi072.stdout:Verifying podman|docker is present... 2024-03-26T01:55:06.555 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T01:55:06.555 INFO:teuthology.orchestra.run.smithi072.stdout:Verifying lvm2 is present... 2024-03-26T01:55:06.555 INFO:teuthology.orchestra.run.smithi072.stdout:Verifying time synchronization is in place... 2024-03-26T01:55:06.562 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T01:55:06.562 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T01:55:06.568 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T01:55:06.568 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout inactive 2024-03-26T01:55:06.574 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout enabled 2024-03-26T01:55:06.580 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout active 2024-03-26T01:55:06.580 INFO:teuthology.orchestra.run.smithi072.stdout:Unit chronyd.service is enabled and running 2024-03-26T01:55:06.580 INFO:teuthology.orchestra.run.smithi072.stdout:Repeating the final host check... 2024-03-26T01:55:06.603 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T01:55:06.603 INFO:teuthology.orchestra.run.smithi072.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-26T01:55:06.603 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl is present 2024-03-26T01:55:06.603 INFO:teuthology.orchestra.run.smithi072.stdout:lvcreate is present 2024-03-26T01:55:06.609 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T01:55:06.609 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T01:55:06.615 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T01:55:06.615 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout inactive 2024-03-26T01:55:06.621 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout enabled 2024-03-26T01:55:06.627 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout active 2024-03-26T01:55:06.627 INFO:teuthology.orchestra.run.smithi072.stdout:Unit chronyd.service is enabled and running 2024-03-26T01:55:06.627 INFO:teuthology.orchestra.run.smithi072.stdout:Host looks OK 2024-03-26T01:55:06.627 INFO:teuthology.orchestra.run.smithi072.stdout:Cluster fsid: c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T01:55:06.627 INFO:teuthology.orchestra.run.smithi072.stdout:Acquiring lock 139842052834928 on /run/cephadm/c86fbdd4-eb13-11ee-95d0-87774f69a715.lock 2024-03-26T01:55:06.627 INFO:teuthology.orchestra.run.smithi072.stdout:Lock 139842052834928 acquired on /run/cephadm/c86fbdd4-eb13-11ee-95d0-87774f69a715.lock 2024-03-26T01:55:06.627 INFO:teuthology.orchestra.run.smithi072.stdout:Verifying IP 172.21.15.72 port 3300 ... 2024-03-26T01:55:06.628 INFO:teuthology.orchestra.run.smithi072.stdout:Verifying IP 172.21.15.72 port 6789 ... 2024-03-26T01:55:06.628 INFO:teuthology.orchestra.run.smithi072.stdout:Base mon IP(s) is [172.21.15.72:3300, 172.21.15.72:6789], mon addrv is [v2:172.21.15.72:3300,v1:172.21.15.72:6789] 2024-03-26T01:55:06.631 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.72 metric 100 2024-03-26T01:55:06.631 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.72 metric 100 2024-03-26T01:55:06.633 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-26T01:55:06.633 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-26T01:55:06.634 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1770sec hoplimit 64 pref medium 2024-03-26T01:55:06.636 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-26T01:55:06.636 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-26T01:55:06.636 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T01:55:06.637 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-26T01:55:06.637 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:b765/64 scope link 2024-03-26T01:55:06.637 INFO:teuthology.orchestra.run.smithi072.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T01:55:06.637 INFO:teuthology.orchestra.run.smithi072.stdout:Mon IP `172.21.15.72` is in CIDR network `172.21.0.0/20` 2024-03-26T01:55:06.637 INFO:teuthology.orchestra.run.smithi072.stdout:Mon IP `172.21.15.72` is in CIDR network `172.21.0.0/20` 2024-03-26T01:55:06.637 INFO:teuthology.orchestra.run.smithi072.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-26T01:55:06.637 INFO:teuthology.orchestra.run.smithi072.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-26T01:55:06.638 INFO:teuthology.orchestra.run.smithi072.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T01:55:07.093 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stdout 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 2024-03-26T01:55:07.093 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T01:55:07.094 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stderr Getting image source signatures 2024-03-26T01:55:07.094 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stderr Copying blob sha256:45ddbf13772d0d6535ecfdcaa58ecfc496e5849512dfbf602c5218332a216bcc 2024-03-26T01:55:07.094 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-26T01:55:07.094 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stderr Copying config sha256:5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 2024-03-26T01:55:07.094 INFO:teuthology.orchestra.run.smithi072.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-26T01:55:08.039 INFO:teuthology.orchestra.run.smithi072.stdout:ceph: stdout ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev) 2024-03-26T01:55:08.039 INFO:teuthology.orchestra.run.smithi072.stdout:Ceph version: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev) 2024-03-26T01:55:08.039 INFO:teuthology.orchestra.run.smithi072.stdout:Extracting ceph user uid/gid from container image... 2024-03-26T01:55:08.943 INFO:teuthology.orchestra.run.smithi072.stdout:stat: stdout 167 167 2024-03-26T01:55:08.943 INFO:teuthology.orchestra.run.smithi072.stdout:Creating initial keys... 2024-03-26T01:55:09.888 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph-authtool: stdout AQD9KgJm8jLGEhAAXYAt5I7gvpWBdyEqbFa4oQ== 2024-03-26T01:55:10.775 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph-authtool: stdout AQD+KgJmyQYsDRAApGT6nXYx5PlQ3+PkmuCryg== 2024-03-26T01:55:11.781 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph-authtool: stdout AQD/KgJm2efhBhAAlkYpPZPa6ck48DWdMnqVeA== 2024-03-26T01:55:11.782 INFO:teuthology.orchestra.run.smithi072.stdout:Creating initial monmap... 2024-03-26T01:55:18.571 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T01:55:18.571 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-26T01:55:18.571 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T01:55:18.571 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T01:55:18.571 INFO:teuthology.orchestra.run.smithi072.stdout:monmaptool for a [v2:172.21.15.72:3300,v1:172.21.15.72:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T01:55:18.572 INFO:teuthology.orchestra.run.smithi072.stdout:setting min_mon_release = quincy 2024-03-26T01:55:18.572 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/monmaptool: set fsid to c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T01:55:18.572 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T01:55:18.572 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:55:18.572 INFO:teuthology.orchestra.run.smithi072.stdout:Creating mon... 2024-03-26T01:55:30.194 INFO:teuthology.orchestra.run.smithi072.stdout:create mon.a on 2024-03-26T01:55:30.399 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-26T01:55:30.537 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-26T01:55:30.722 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715.target. 2024-03-26T01:55:30.723 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715.target. 2024-03-26T01:55:30.957 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@mon.a 2024-03-26T01:55:30.957 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-26T01:55:31.114 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715.target.wants/ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@.service. 2024-03-26T01:55:31.518 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:31 smithi072 podman[29646]: 2024-03-26 01:55:31.244478507 +0000 UTC m=+0.017929473 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T01:55:31.518 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:31 smithi072 podman[29646]: 2024-03-26 01:55:31.518558959 +0000 UTC m=+0.292009919 container init f344fc3cb321567a6481d98312fc8601f2971d3e6c971206bddd215b589daca4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T01:55:31.624 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T01:55:31.624 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout disabled 2024-03-26T01:55:31.630 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T01:55:31.630 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout inactive 2024-03-26T01:55:31.630 INFO:teuthology.orchestra.run.smithi072.stdout:firewalld.service is not enabled 2024-03-26T01:55:31.630 INFO:teuthology.orchestra.run.smithi072.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T01:55:31.631 INFO:teuthology.orchestra.run.smithi072.stdout:Waiting for mon to start... 2024-03-26T01:55:31.631 INFO:teuthology.orchestra.run.smithi072.stdout:Waiting for mon... 2024-03-26T01:55:31.859 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:31 smithi072 podman[29646]: 2024-03-26 01:55:31.523719543 +0000 UTC m=+0.297170504 container start f344fc3cb321567a6481d98312fc8601f2971d3e6c971206bddd215b589daca4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-a, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-26T01:55:31.859 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:31 smithi072 ceph-mon[29684]: mkfs c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T01:55:31.859 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:31 smithi072 ceph-mon[29684]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T01:55:31.859 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:31 smithi072 bash[29646]: f344fc3cb321567a6481d98312fc8601f2971d3e6c971206bddd215b589daca4 2024-03-26T01:55:31.859 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:31 smithi072 systemd[1]: Started Ceph mon.a for c86fbdd4-eb13-11ee-95d0-87774f69a715. 2024-03-26T01:55:32.695 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout cluster: 2024-03-26T01:55:32.695 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout id: c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T01:55:32.695 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-26T01:55:32.695 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-03-26T01:55:32.695 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout services: 2024-03-26T01:55:32.695 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.57694s) 2024-03-26T01:55:32.695 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-26T01:55:32.695 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-26T01:55:32.695 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-03-26T01:55:32.696 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout data: 2024-03-26T01:55:32.696 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-26T01:55:32.696 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-26T01:55:32.696 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-26T01:55:32.696 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout pgs: 2024-03-26T01:55:32.696 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-03-26T01:55:32.696 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.086+0000 7fe3679d0700 1 Processor -- start 2024-03-26T01:55:32.696 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.086+0000 7fe3679d0700 1 -- start start 2024-03-26T01:55:32.696 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.086+0000 7fe3679d0700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe36007bfe0 0x7fe36007a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:55:32.696 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.086+0000 7fe3679d0700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36007aa10 con 0x7fe36007bfe0 2024-03-26T01:55:32.696 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.086+0000 7fe36576c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe36007bfe0 0x7fe36007a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:55:32.696 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.086+0000 7fe36576c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe36007bfe0 0x7fe36007a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55686/0 (socket says 172.21.15.72:55686) 2024-03-26T01:55:32.696 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.086+0000 7fe36576c700 1 -- 172.21.15.72:0/144766403 learned_addr learned my addr 172.21.15.72:0/144766403 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:55:32.696 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.087+0000 7fe36576c700 1 -- 172.21.15.72:0/144766403 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36007ab50 con 0x7fe36007bfe0 2024-03-26T01:55:32.697 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.087+0000 7fe36576c700 1 --2- 172.21.15.72:0/144766403 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe36007bfe0 0x7fe36007a440 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe35c0098a0 tx=0x7fe35c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4f75d49e441a2cd server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:55:32.697 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.087+0000 7fe357fff700 1 -- 172.21.15.72:0/144766403 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe35c004160 con 0x7fe36007bfe0 2024-03-26T01:55:32.697 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.087+0000 7fe357fff700 1 -- 172.21.15.72:0/144766403 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35c0042c0 con 0x7fe36007bfe0 2024-03-26T01:55:32.697 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.087+0000 7fe357fff700 1 -- 172.21.15.72:0/144766403 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe35c0045a0 con 0x7fe36007bfe0 2024-03-26T01:55:32.697 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.088+0000 7fe3679d0700 1 -- 172.21.15.72:0/144766403 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe36007bfe0 msgr2=0x7fe36007a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:55:32.697 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.088+0000 7fe3679d0700 1 --2- 172.21.15.72:0/144766403 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe36007bfe0 0x7fe36007a440 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe35c0098a0 tx=0x7fe35c009bb0 comp rx=0 tx=0).stop 2024-03-26T01:55:32.697 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.088+0000 7fe3679d0700 1 -- 172.21.15.72:0/144766403 shutdown_connections 2024-03-26T01:55:32.697 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.088+0000 7fe3679d0700 1 --2- 172.21.15.72:0/144766403 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe36007bfe0 0x7fe36007a440 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:32.697 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.088+0000 7fe3679d0700 1 -- 172.21.15.72:0/144766403 >> 172.21.15.72:0/144766403 conn(0x7fe360101680 msgr2=0x7fe360103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:55:32.697 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.088+0000 7fe3679d0700 1 -- 172.21.15.72:0/144766403 shutdown_connections 2024-03-26T01:55:32.697 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.088+0000 7fe3679d0700 1 -- 172.21.15.72:0/144766403 wait complete. 2024-03-26T01:55:32.697 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.088+0000 7fe3679d0700 1 Processor -- start 2024-03-26T01:55:32.697 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.088+0000 7fe3679d0700 1 -- start start 2024-03-26T01:55:32.697 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.089+0000 7fe3679d0700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe36007bfe0 0x7fe36019fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:55:32.697 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.089+0000 7fe3679d0700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3601a0160 con 0x7fe36007bfe0 2024-03-26T01:55:32.698 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.089+0000 7fe36576c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe36007bfe0 0x7fe36019fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:55:32.698 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.089+0000 7fe36576c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe36007bfe0 0x7fe36019fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55696/0 (socket says 172.21.15.72:55696) 2024-03-26T01:55:32.698 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.089+0000 7fe36576c700 1 -- 172.21.15.72:0/2724586363 learned_addr learned my addr 172.21.15.72:0/2724586363 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:55:32.698 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.089+0000 7fe36576c700 1 -- 172.21.15.72:0/2724586363 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe35c009550 con 0x7fe36007bfe0 2024-03-26T01:55:32.698 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.089+0000 7fe36576c700 1 --2- 172.21.15.72:0/2724586363 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe36007bfe0 0x7fe36019fc20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe35c004000 tx=0x7fe35c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:55:32.698 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.089+0000 7fe3567fc700 1 -- 172.21.15.72:0/2724586363 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe35c011630 con 0x7fe36007bfe0 2024-03-26T01:55:32.698 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.089+0000 7fe3567fc700 1 -- 172.21.15.72:0/2724586363 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe35c011790 con 0x7fe36007bfe0 2024-03-26T01:55:32.698 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.089+0000 7fe3567fc700 1 -- 172.21.15.72:0/2724586363 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe35c011a30 con 0x7fe36007bfe0 2024-03-26T01:55:32.698 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.089+0000 7fe3679d0700 1 -- 172.21.15.72:0/2724586363 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3601a0360 con 0x7fe36007bfe0 2024-03-26T01:55:32.698 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.089+0000 7fe3679d0700 1 -- 172.21.15.72:0/2724586363 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3601a0800 con 0x7fe36007bfe0 2024-03-26T01:55:32.698 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.089+0000 7fe3567fc700 1 -- 172.21.15.72:0/2724586363 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fe35c022070 con 0x7fe36007bfe0 2024-03-26T01:55:32.698 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.089+0000 7fe3567fc700 1 -- 172.21.15.72:0/2724586363 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fe35c01c450 con 0x7fe36007bfe0 2024-03-26T01:55:32.698 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.090+0000 7fe3679d0700 1 -- 172.21.15.72:0/2724586363 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe360199b80 con 0x7fe36007bfe0 2024-03-26T01:55:32.699 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.092+0000 7fe3567fc700 1 -- 172.21.15.72:0/2724586363 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fe35c011b90 con 0x7fe36007bfe0 2024-03-26T01:55:32.699 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.162+0000 7fe3679d0700 1 -- 172.21.15.72:0/2724586363 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fe36007aec0 con 0x7fe36007bfe0 2024-03-26T01:55:32.699 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.162+0000 7fe3567fc700 1 -- 172.21.15.72:0/2724586363 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+316 (secure 0 0 0) 0x7fe35c011d30 con 0x7fe36007bfe0 2024-03-26T01:55:32.699 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.163+0000 7fe3679d0700 1 -- 172.21.15.72:0/2724586363 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe36007bfe0 msgr2=0x7fe36019fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:55:32.699 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.163+0000 7fe3679d0700 1 --2- 172.21.15.72:0/2724586363 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe36007bfe0 0x7fe36019fc20 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe35c004000 tx=0x7fe35c004750 comp rx=0 tx=0).stop 2024-03-26T01:55:32.699 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.163+0000 7fe3679d0700 1 -- 172.21.15.72:0/2724586363 shutdown_connections 2024-03-26T01:55:32.699 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.163+0000 7fe3679d0700 1 --2- 172.21.15.72:0/2724586363 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe36007bfe0 0x7fe36019fc20 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:32.699 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.163+0000 7fe3679d0700 1 -- 172.21.15.72:0/2724586363 >> 172.21.15.72:0/2724586363 conn(0x7fe360101680 msgr2=0x7fe360103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:55:32.699 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.163+0000 7fe3679d0700 1 -- 172.21.15.72:0/2724586363 shutdown_connections 2024-03-26T01:55:32.699 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:32.163+0000 7fe3679d0700 1 -- 172.21.15.72:0/2724586363 wait complete. 2024-03-26T01:55:32.699 INFO:teuthology.orchestra.run.smithi072.stdout:mon is available 2024-03-26T01:55:32.699 INFO:teuthology.orchestra.run.smithi072.stdout:Assimilating anything we can from ceph.conf... 2024-03-26T01:55:32.718 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:32 smithi072 ceph-mon[29684]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T01:55:32.718 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:32 smithi072 ceph-mon[29684]: monmap e1: 1 mons at {a=[v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T01:55:32.718 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:32 smithi072 ceph-mon[29684]: fsmap 2024-03-26T01:55:32.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:32 smithi072 ceph-mon[29684]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T01:55:32.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:32 smithi072 ceph-mon[29684]: mgrmap e1: no daemons active 2024-03-26T01:55:32.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:32 smithi072 ceph-mon[29684]: from='client.? 172.21.15.72:0/2724586363' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-26T01:55:33.843 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:33 smithi072 ceph-mon[29684]: from='client.? 172.21.15.72:0/1055711816' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T01:55:33.843 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:33 smithi072 ceph-mon[29684]: from='client.? 172.21.15.72:0/1055711816' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-26T01:55:33.846 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-03-26T01:55:33.852 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T01:55:33.852 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout fsid = c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T01:55:33.852 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.72:3300,v1:172.21.15.72:6789] 2024-03-26T01:55:33.852 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T01:55:33.852 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T01:55:33.852 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T01:55:33.852 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T01:55:33.852 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-03-26T01:55:33.852 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T01:55:33.852 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T01:55:33.853 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-03-26T01:55:33.853 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T01:55:33.853 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T01:55:33.853 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T01:55:33.853 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.169+0000 7f01a00e3700 1 Processor -- start 2024-03-26T01:55:33.853 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.169+0000 7f01a00e3700 1 -- start start 2024-03-26T01:55:33.853 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.170+0000 7f01a00e3700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01981060f0 0x7f0198106510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:55:33.853 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.170+0000 7f01a00e3700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0198106ae0 con 0x7f01981060f0 2024-03-26T01:55:33.853 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.170+0000 7f019de7f700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01981060f0 0x7f0198106510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:55:33.853 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.170+0000 7f019de7f700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01981060f0 0x7f0198106510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55710/0 (socket says 172.21.15.72:55710) 2024-03-26T01:55:33.853 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.170+0000 7f019de7f700 1 -- 172.21.15.72:0/1670762301 learned_addr learned my addr 172.21.15.72:0/1670762301 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:55:33.853 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.170+0000 7f019de7f700 1 -- 172.21.15.72:0/1670762301 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01981072f0 con 0x7f01981060f0 2024-03-26T01:55:33.853 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.170+0000 7f019de7f700 1 --2- 172.21.15.72:0/1670762301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01981060f0 0x7f0198106510 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f01880098a0 tx=0x7f0188009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b011fbb419029fe server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:55:33.853 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.170+0000 7f019ce7d700 1 -- 172.21.15.72:0/1670762301 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0188004160 con 0x7f01981060f0 2024-03-26T01:55:33.854 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.170+0000 7f019ce7d700 1 -- 172.21.15.72:0/1670762301 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01880042c0 con 0x7f01981060f0 2024-03-26T01:55:33.854 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.170+0000 7f019ce7d700 1 -- 172.21.15.72:0/1670762301 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01880045a0 con 0x7f01981060f0 2024-03-26T01:55:33.854 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.171+0000 7f01a00e3700 1 -- 172.21.15.72:0/1670762301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01981060f0 msgr2=0x7f0198106510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:55:33.854 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.171+0000 7f01a00e3700 1 --2- 172.21.15.72:0/1670762301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01981060f0 0x7f0198106510 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f01880098a0 tx=0x7f0188009bb0 comp rx=0 tx=0).stop 2024-03-26T01:55:33.854 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.171+0000 7f01a00e3700 1 -- 172.21.15.72:0/1670762301 shutdown_connections 2024-03-26T01:55:33.854 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.171+0000 7f01a00e3700 1 --2- 172.21.15.72:0/1670762301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01981060f0 0x7f0198106510 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:33.854 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.171+0000 7f01a00e3700 1 -- 172.21.15.72:0/1670762301 >> 172.21.15.72:0/1670762301 conn(0x7f01981016b0 msgr2=0x7f0198103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:55:33.854 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.171+0000 7f01a00e3700 1 -- 172.21.15.72:0/1670762301 shutdown_connections 2024-03-26T01:55:33.854 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.171+0000 7f01a00e3700 1 -- 172.21.15.72:0/1670762301 wait complete. 2024-03-26T01:55:33.854 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.171+0000 7f01a00e3700 1 Processor -- start 2024-03-26T01:55:33.854 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.172+0000 7f01a00e3700 1 -- start start 2024-03-26T01:55:33.854 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.172+0000 7f01a00e3700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01981060f0 0x7f019819b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:55:33.854 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.172+0000 7f01a00e3700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019819bd50 con 0x7f01981060f0 2024-03-26T01:55:33.854 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.172+0000 7f019de7f700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01981060f0 0x7f019819b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:55:33.855 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.172+0000 7f019de7f700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01981060f0 0x7f019819b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55724/0 (socket says 172.21.15.72:55724) 2024-03-26T01:55:33.855 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.172+0000 7f019de7f700 1 -- 172.21.15.72:0/1055711816 learned_addr learned my addr 172.21.15.72:0/1055711816 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:55:33.855 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.172+0000 7f019de7f700 1 -- 172.21.15.72:0/1055711816 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0188009550 con 0x7f01981060f0 2024-03-26T01:55:33.855 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.172+0000 7f019de7f700 1 --2- 172.21.15.72:0/1055711816 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01981060f0 0x7f019819b810 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0198106d80 tx=0x7f0188004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:55:33.855 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.172+0000 7f018effd700 1 -- 172.21.15.72:0/1055711816 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0188003da0 con 0x7f01981060f0 2024-03-26T01:55:33.855 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.172+0000 7f018effd700 1 -- 172.21.15.72:0/1055711816 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0188011630 con 0x7f01981060f0 2024-03-26T01:55:33.855 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.172+0000 7f01a00e3700 1 -- 172.21.15.72:0/1055711816 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f019819bf50 con 0x7f01981060f0 2024-03-26T01:55:33.855 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.172+0000 7f018effd700 1 -- 172.21.15.72:0/1055711816 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0188011910 con 0x7f01981060f0 2024-03-26T01:55:33.855 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.172+0000 7f01a00e3700 1 -- 172.21.15.72:0/1055711816 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f019819c3f0 con 0x7f01981060f0 2024-03-26T01:55:33.855 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.173+0000 7f018effd700 1 -- 172.21.15.72:0/1055711816 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f018801c460 con 0x7f01981060f0 2024-03-26T01:55:33.855 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.173+0000 7f018effd700 1 -- 172.21.15.72:0/1055711816 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f018801aba0 con 0x7f01981060f0 2024-03-26T01:55:33.855 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.173+0000 7f01a00e3700 1 -- 172.21.15.72:0/1055711816 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0198107660 con 0x7f01981060f0 2024-03-26T01:55:33.855 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.175+0000 7f018effd700 1 -- 172.21.15.72:0/1055711816 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f0188011e50 con 0x7f01981060f0 2024-03-26T01:55:33.855 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.244+0000 7f01a00e3700 1 -- 172.21.15.72:0/1055711816 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f0198107840 con 0x7f01981060f0 2024-03-26T01:55:33.856 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.246+0000 7f018effd700 1 -- 172.21.15.72:0/1055711816 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+338 (secure 0 0 0) 0x7f018801ae00 con 0x7f01981060f0 2024-03-26T01:55:33.856 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.247+0000 7f018effd700 1 -- 172.21.15.72:0/1055711816 <== mon.0 v2:172.21.15.72:3300/0 8 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f0188036930 con 0x7f01981060f0 2024-03-26T01:55:33.856 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.248+0000 7f01a00e3700 1 -- 172.21.15.72:0/1055711816 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01981060f0 msgr2=0x7f019819b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:55:33.856 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.248+0000 7f01a00e3700 1 --2- 172.21.15.72:0/1055711816 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01981060f0 0x7f019819b810 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f0198106d80 tx=0x7f0188004750 comp rx=0 tx=0).stop 2024-03-26T01:55:33.856 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.248+0000 7f01a00e3700 1 -- 172.21.15.72:0/1055711816 shutdown_connections 2024-03-26T01:55:33.856 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.248+0000 7f01a00e3700 1 --2- 172.21.15.72:0/1055711816 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01981060f0 0x7f019819b810 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:33.856 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.248+0000 7f01a00e3700 1 -- 172.21.15.72:0/1055711816 >> 172.21.15.72:0/1055711816 conn(0x7f01981016b0 msgr2=0x7f019807b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:55:33.856 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.248+0000 7f01a00e3700 1 -- 172.21.15.72:0/1055711816 shutdown_connections 2024-03-26T01:55:33.856 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:33.248+0000 7f01a00e3700 1 -- 172.21.15.72:0/1055711816 wait complete. 2024-03-26T01:55:33.856 INFO:teuthology.orchestra.run.smithi072.stdout:Generating new minimal ceph.conf... 2024-03-26T01:55:34.968 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:34 smithi072 ceph-mon[29684]: from='client.? 172.21.15.72:0/1488297219' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:55:35.148 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.281+0000 7f54e112a700 1 Processor -- start 2024-03-26T01:55:35.148 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.281+0000 7f54e112a700 1 -- start start 2024-03-26T01:55:35.148 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.281+0000 7f54e112a700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54dc105000 0x7f54dc105420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:55:35.148 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.281+0000 7f54e112a700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc1059f0 con 0x7f54dc105000 2024-03-26T01:55:35.148 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.282+0000 7f54dad9d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54dc105000 0x7f54dc105420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:55:35.148 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.282+0000 7f54dad9d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54dc105000 0x7f54dc105420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55736/0 (socket says 172.21.15.72:55736) 2024-03-26T01:55:35.149 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.282+0000 7f54dad9d700 1 -- 172.21.15.72:0/2271809775 learned_addr learned my addr 172.21.15.72:0/2271809775 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:55:35.149 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.282+0000 7f54dad9d700 1 -- 172.21.15.72:0/2271809775 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54dc106200 con 0x7f54dc105000 2024-03-26T01:55:35.149 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.282+0000 7f54dad9d700 1 --2- 172.21.15.72:0/2271809775 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54dc105000 0x7f54dc105420 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f54cc0098a0 tx=0x7f54cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98c5b5af5521bb7b server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:55:35.149 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.282+0000 7f54d9d9b700 1 -- 172.21.15.72:0/2271809775 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54cc004160 con 0x7f54dc105000 2024-03-26T01:55:35.149 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.282+0000 7f54d9d9b700 1 -- 172.21.15.72:0/2271809775 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f54cc0042c0 con 0x7f54dc105000 2024-03-26T01:55:35.149 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.282+0000 7f54d9d9b700 1 -- 172.21.15.72:0/2271809775 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54cc0119b0 con 0x7f54dc105000 2024-03-26T01:55:35.149 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.283+0000 7f54e112a700 1 -- 172.21.15.72:0/2271809775 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54dc105000 msgr2=0x7f54dc105420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:55:35.149 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.283+0000 7f54e112a700 1 --2- 172.21.15.72:0/2271809775 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54dc105000 0x7f54dc105420 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f54cc0098a0 tx=0x7f54cc009bb0 comp rx=0 tx=0).stop 2024-03-26T01:55:35.149 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.283+0000 7f54e112a700 1 -- 172.21.15.72:0/2271809775 shutdown_connections 2024-03-26T01:55:35.149 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.283+0000 7f54e112a700 1 --2- 172.21.15.72:0/2271809775 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54dc105000 0x7f54dc105420 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:35.149 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.283+0000 7f54e112a700 1 -- 172.21.15.72:0/2271809775 >> 172.21.15.72:0/2271809775 conn(0x7f54dc1005a0 msgr2=0x7f54dc1029e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:55:35.149 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.283+0000 7f54e112a700 1 -- 172.21.15.72:0/2271809775 shutdown_connections 2024-03-26T01:55:35.149 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.283+0000 7f54e112a700 1 -- 172.21.15.72:0/2271809775 wait complete. 2024-03-26T01:55:35.150 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.284+0000 7f54e112a700 1 Processor -- start 2024-03-26T01:55:35.150 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.284+0000 7f54e112a700 1 -- start start 2024-03-26T01:55:35.150 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.284+0000 7f54e112a700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54dc105000 0x7f54dc079b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:55:35.150 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.284+0000 7f54e112a700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc19cc00 con 0x7f54dc105000 2024-03-26T01:55:35.150 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.284+0000 7f54dad9d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54dc105000 0x7f54dc079b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:55:35.150 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.284+0000 7f54dad9d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54dc105000 0x7f54dc079b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55738/0 (socket says 172.21.15.72:55738) 2024-03-26T01:55:35.150 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.284+0000 7f54dad9d700 1 -- 172.21.15.72:0/1488297219 learned_addr learned my addr 172.21.15.72:0/1488297219 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:55:35.150 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.284+0000 7f54dad9d700 1 -- 172.21.15.72:0/1488297219 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54cc009550 con 0x7f54dc105000 2024-03-26T01:55:35.150 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.284+0000 7f54dad9d700 1 --2- 172.21.15.72:0/1488297219 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54dc105000 0x7f54dc079b70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f54cc006af0 tx=0x7f54cc011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:55:35.150 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.285+0000 7f54c3fff700 1 -- 172.21.15.72:0/1488297219 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54cc0039e0 con 0x7f54dc105000 2024-03-26T01:55:35.150 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.285+0000 7f54c3fff700 1 -- 172.21.15.72:0/1488297219 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f54cc003b40 con 0x7f54dc105000 2024-03-26T01:55:35.150 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.285+0000 7f54e112a700 1 -- 172.21.15.72:0/1488297219 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54dc19cd40 con 0x7f54dc105000 2024-03-26T01:55:35.150 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.285+0000 7f54c3fff700 1 -- 172.21.15.72:0/1488297219 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54cc019720 con 0x7f54dc105000 2024-03-26T01:55:35.151 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.285+0000 7f54e112a700 1 -- 172.21.15.72:0/1488297219 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54dc19d120 con 0x7f54dc105000 2024-03-26T01:55:35.151 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.285+0000 7f54c3fff700 1 -- 172.21.15.72:0/1488297219 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f54cc019d50 con 0x7f54dc105000 2024-03-26T01:55:35.151 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.285+0000 7f54c3fff700 1 -- 172.21.15.72:0/1488297219 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f54cc02c6f0 con 0x7f54dc105000 2024-03-26T01:55:35.151 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.286+0000 7f54e112a700 1 -- 172.21.15.72:0/1488297219 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54dc0636b0 con 0x7f54dc105000 2024-03-26T01:55:35.151 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.288+0000 7f54c3fff700 1 -- 172.21.15.72:0/1488297219 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f54cc042dd0 con 0x7f54dc105000 2024-03-26T01:55:35.151 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.368+0000 7f54e112a700 1 -- 172.21.15.72:0/1488297219 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f54dc19d3d0 con 0x7f54dc105000 2024-03-26T01:55:35.151 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.368+0000 7f54c3fff700 1 -- 172.21.15.72:0/1488297219 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f54cc031020 con 0x7f54dc105000 2024-03-26T01:55:35.151 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.369+0000 7f54e112a700 1 -- 172.21.15.72:0/1488297219 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54dc105000 msgr2=0x7f54dc079b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:55:35.151 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.369+0000 7f54e112a700 1 --2- 172.21.15.72:0/1488297219 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54dc105000 0x7f54dc079b70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f54cc006af0 tx=0x7f54cc011a50 comp rx=0 tx=0).stop 2024-03-26T01:55:35.151 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.369+0000 7f54e112a700 1 -- 172.21.15.72:0/1488297219 shutdown_connections 2024-03-26T01:55:35.151 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.369+0000 7f54e112a700 1 --2- 172.21.15.72:0/1488297219 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f54dc105000 0x7f54dc079b70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:35.151 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.369+0000 7f54e112a700 1 -- 172.21.15.72:0/1488297219 >> 172.21.15.72:0/1488297219 conn(0x7f54dc1005a0 msgr2=0x7f54dc101100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:55:35.151 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.370+0000 7f54e112a700 1 -- 172.21.15.72:0/1488297219 shutdown_connections 2024-03-26T01:55:35.152 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:34.370+0000 7f54e112a700 1 -- 172.21.15.72:0/1488297219 wait complete. 2024-03-26T01:55:35.152 INFO:teuthology.orchestra.run.smithi072.stdout:Restarting the monitor... 2024-03-26T01:55:35.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:35 smithi072 systemd[1]: Stopping Ceph mon.a for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T01:55:37.718 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:37 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-a[29661]: 2024-03-26T01:55:37.431+0000 7fd9c19b6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T01:55:37.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:37 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-a[29661]: 2024-03-26T01:55:37.431+0000 7fd9c19b6700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-26T01:55:37.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:37 smithi072 podman[30016]: 2024-03-26 01:55:37.452236247 +0000 UTC m=+2.138585904 container died f344fc3cb321567a6481d98312fc8601f2971d3e6c971206bddd215b589daca4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-a, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-26T01:55:38.174 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:37 smithi072 podman[30016]: 2024-03-26 01:55:37.922293385 +0000 UTC m=+2.608643051 container cleanup f344fc3cb321567a6481d98312fc8601f2971d3e6c971206bddd215b589daca4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T01:55:38.174 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:37 smithi072 bash[30016]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-a 2024-03-26T01:55:38.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 podman[30034]: 2024-03-26 01:55:38.1744898 +0000 UTC m=+0.715447316 container remove f344fc3cb321567a6481d98312fc8601f2971d3e6c971206bddd215b589daca4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-26T01:55:38.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 systemd[1]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-26T01:55:38.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 systemd[1]: Stopped Ceph mon.a for c86fbdd4-eb13-11ee-95d0-87774f69a715. 2024-03-26T01:55:38.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 systemd[1]: Starting Ceph mon.a for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T01:55:38.748 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 podman[30139]: 2024-03-26 01:55:38.484635978 +0000 UTC m=+0.106363029 container create 52553733e232008acdda40c51484e26e7561b37ee6b2680a5a5090858c8021d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5) 2024-03-26T01:55:38.748 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 podman[30139]: 2024-03-26 01:55:38.395821403 +0000 UTC m=+0.017548417 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T01:55:38.748 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 podman[30139]: 2024-03-26 01:55:38.671530515 +0000 UTC m=+0.293257521 container init 52553733e232008acdda40c51484e26e7561b37ee6b2680a5a5090858c8021d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-26T01:55:38.748 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 podman[30139]: 2024-03-26 01:55:38.676920519 +0000 UTC m=+0.298647528 container start 52553733e232008acdda40c51484e26e7561b37ee6b2680a5a5090858c8021d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-26T01:55:38.748 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T01:55:38.749 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 7 2024-03-26T01:55:38.749 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: pidfile_write: ignore empty --pid-file 2024-03-26T01:55:38.749 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: load: jerasure load: lrc 2024-03-26T01:55:38.749 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: RocksDB version: 7.9.2 2024-03-26T01:55:38.749 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Git sha 0 2024-03-26T01:55:38.749 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T01:55:38.749 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: DB SUMMARY 2024-03-26T01:55:38.749 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: DB Session ID: OKBH2VZV0R6CC88Y8LC3 2024-03-26T01:55:38.749 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: CURRENT file: CURRENT 2024-03-26T01:55:38.749 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T01:55:38.749 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-26T01:55:38.749 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-26T01:55:38.750 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 87703 ; 2024-03-26T01:55:38.750 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.error_if_exists: 0 2024-03-26T01:55:38.750 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.create_if_missing: 0 2024-03-26T01:55:38.750 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.paranoid_checks: 1 2024-03-26T01:55:38.750 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T01:55:38.750 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T01:55:38.750 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T01:55:38.750 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.env: 0x56265d925ae0 2024-03-26T01:55:38.750 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T01:55:38.750 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.info_log: 0x56265f93f920 2024-03-26T01:55:38.750 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T01:55:38.750 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.statistics: (nil) 2024-03-26T01:55:38.750 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.use_fsync: 0 2024-03-26T01:55:38.750 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.max_log_file_size: 0 2024-03-26T01:55:38.751 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T01:55:38.751 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T01:55:38.751 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T01:55:38.751 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T01:55:38.751 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.allow_fallocate: 1 2024-03-26T01:55:38.751 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T01:55:38.751 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T01:55:38.751 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.use_direct_reads: 0 2024-03-26T01:55:38.751 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T01:55:38.751 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T01:55:38.751 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.db_log_dir: 2024-03-26T01:55:38.751 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.wal_dir: 2024-03-26T01:55:38.751 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T01:55:38.751 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T01:55:38.751 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T01:55:38.751 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T01:55:38.752 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T01:55:38.752 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T01:55:38.752 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T01:55:38.752 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T01:55:38.752 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.write_buffer_manager: 0x56265f9121e0 2024-03-26T01:55:38.752 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T01:55:38.752 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T01:55:38.752 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T01:55:38.752 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T01:55:38.752 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T01:55:38.752 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T01:55:38.753 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T01:55:38.753 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T01:55:38.753 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.unordered_write: 0 2024-03-26T01:55:38.754 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T01:55:38.754 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T01:55:38.754 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T01:55:38.754 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T01:55:38.754 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.row_cache: None 2024-03-26T01:55:38.754 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.wal_filter: None 2024-03-26T01:55:38.754 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T01:55:38.754 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T01:55:38.754 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.two_write_queues: 0 2024-03-26T01:55:38.754 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T01:55:38.754 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.wal_compression: 0 2024-03-26T01:55:38.754 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.atomic_flush: 0 2024-03-26T01:55:38.755 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T01:55:38.755 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T01:55:38.755 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T01:55:38.755 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.log_readahead_size: 0 2024-03-26T01:55:38.755 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T01:55:38.755 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T01:55:38.755 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T01:55:38.755 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T01:55:38.755 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T01:55:38.755 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T01:55:38.755 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T01:55:38.755 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.max_background_jobs: 2 2024-03-26T01:55:38.755 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.max_background_compactions: -1 2024-03-26T01:55:38.755 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.max_subcompactions: 1 2024-03-26T01:55:38.755 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T01:55:38.756 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T01:55:38.756 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T01:55:38.756 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T01:55:38.756 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T01:55:38.756 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T01:55:38.756 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T01:55:38.756 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T01:55:38.756 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.max_open_files: -1 2024-03-26T01:55:38.756 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T01:55:38.756 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T01:55:38.756 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T01:55:38.756 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T01:55:38.756 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.max_background_flushes: -1 2024-03-26T01:55:38.756 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Compression algorithms supported: 2024-03-26T01:55:38.756 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T01:55:38.757 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: kZSTD supported: 0 2024-03-26T01:55:38.757 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: kXpressCompression supported: 0 2024-03-26T01:55:38.757 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T01:55:38.757 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: kZlibCompression supported: 1 2024-03-26T01:55:38.757 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: kSnappyCompression supported: 1 2024-03-26T01:55:38.757 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: kLZ4Compression supported: 1 2024-03-26T01:55:38.757 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: kBZip2Compression supported: 0 2024-03-26T01:55:38.757 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T01:55:38.757 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T01:55:38.757 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-26T01:55:38.757 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T01:55:38.757 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T01:55:38.757 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.merge_operator: 2024-03-26T01:55:38.757 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.compaction_filter: None 2024-03-26T01:55:38.758 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T01:55:38.758 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T01:55:38.758 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T01:55:38.758 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T01:55:38.758 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56265f93fa20) 2024-03-26T01:55:38.758 INFO:journalctl@ceph.mon.a.smithi072.stdout: cache_index_and_filter_blocks: 1 2024-03-26T01:55:38.758 INFO:journalctl@ceph.mon.a.smithi072.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T01:55:38.758 INFO:journalctl@ceph.mon.a.smithi072.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T01:55:38.758 INFO:journalctl@ceph.mon.a.smithi072.stdout: pin_top_level_index_and_filter: 1 2024-03-26T01:55:38.758 INFO:journalctl@ceph.mon.a.smithi072.stdout: index_type: 0 2024-03-26T01:55:38.758 INFO:journalctl@ceph.mon.a.smithi072.stdout: data_block_index_type: 0 2024-03-26T01:55:38.758 INFO:journalctl@ceph.mon.a.smithi072.stdout: index_shortening: 1 2024-03-26T01:55:38.758 INFO:journalctl@ceph.mon.a.smithi072.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T01:55:38.758 INFO:journalctl@ceph.mon.a.smithi072.stdout: checksum: 4 2024-03-26T01:55:38.758 INFO:journalctl@ceph.mon.a.smithi072.stdout: no_block_cache: 0 2024-03-26T01:55:38.759 INFO:journalctl@ceph.mon.a.smithi072.stdout: block_cache: 0x56265f943090 2024-03-26T01:55:38.759 INFO:journalctl@ceph.mon.a.smithi072.stdout: block_cache_name: BinnedLRUCache 2024-03-26T01:55:38.759 INFO:journalctl@ceph.mon.a.smithi072.stdout: block_cache_options: 2024-03-26T01:55:38.759 INFO:journalctl@ceph.mon.a.smithi072.stdout: capacity : 536870912 2024-03-26T01:55:38.759 INFO:journalctl@ceph.mon.a.smithi072.stdout: num_shard_bits : 4 2024-03-26T01:55:38.759 INFO:journalctl@ceph.mon.a.smithi072.stdout: strict_capacity_limit : 0 2024-03-26T01:55:38.759 INFO:journalctl@ceph.mon.a.smithi072.stdout: high_pri_pool_ratio: 0.000 2024-03-26T01:55:38.759 INFO:journalctl@ceph.mon.a.smithi072.stdout: block_cache_compressed: (nil) 2024-03-26T01:55:38.759 INFO:journalctl@ceph.mon.a.smithi072.stdout: persistent_cache: (nil) 2024-03-26T01:55:38.759 INFO:journalctl@ceph.mon.a.smithi072.stdout: block_size: 4096 2024-03-26T01:55:38.759 INFO:journalctl@ceph.mon.a.smithi072.stdout: block_size_deviation: 10 2024-03-26T01:55:38.759 INFO:journalctl@ceph.mon.a.smithi072.stdout: block_restart_interval: 16 2024-03-26T01:55:38.759 INFO:journalctl@ceph.mon.a.smithi072.stdout: index_block_restart_interval: 1 2024-03-26T01:55:38.759 INFO:journalctl@ceph.mon.a.smithi072.stdout: metadata_block_size: 4096 2024-03-26T01:55:38.760 INFO:journalctl@ceph.mon.a.smithi072.stdout: partition_filters: 0 2024-03-26T01:55:38.760 INFO:journalctl@ceph.mon.a.smithi072.stdout: use_delta_encoding: 1 2024-03-26T01:55:38.760 INFO:journalctl@ceph.mon.a.smithi072.stdout: filter_policy: bloomfilter 2024-03-26T01:55:38.760 INFO:journalctl@ceph.mon.a.smithi072.stdout: whole_key_filtering: 1 2024-03-26T01:55:38.760 INFO:journalctl@ceph.mon.a.smithi072.stdout: verify_compression: 0 2024-03-26T01:55:38.760 INFO:journalctl@ceph.mon.a.smithi072.stdout: read_amp_bytes_per_bit: 0 2024-03-26T01:55:38.760 INFO:journalctl@ceph.mon.a.smithi072.stdout: format_version: 5 2024-03-26T01:55:38.760 INFO:journalctl@ceph.mon.a.smithi072.stdout: enable_index_compression: 1 2024-03-26T01:55:38.760 INFO:journalctl@ceph.mon.a.smithi072.stdout: block_align: 0 2024-03-26T01:55:38.760 INFO:journalctl@ceph.mon.a.smithi072.stdout: max_auto_readahead_size: 262144 2024-03-26T01:55:38.760 INFO:journalctl@ceph.mon.a.smithi072.stdout: prepopulate_block_cache: 0 2024-03-26T01:55:38.760 INFO:journalctl@ceph.mon.a.smithi072.stdout: initial_auto_readahead_size: 8192 2024-03-26T01:55:38.760 INFO:journalctl@ceph.mon.a.smithi072.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T01:55:38.760 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T01:55:38.761 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T01:55:38.761 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.compression: NoCompression 2024-03-26T01:55:38.761 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T01:55:38.761 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T01:55:38.761 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T01:55:38.761 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.num_levels: 7 2024-03-26T01:55:38.761 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T01:55:38.761 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T01:55:38.761 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T01:55:38.761 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T01:55:38.761 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T01:55:38.761 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T01:55:38.761 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T01:55:38.761 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T01:55:38.761 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T01:55:38.762 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T01:55:38.762 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T01:55:38.762 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T01:55:38.762 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T01:55:38.762 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T01:55:38.762 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T01:55:38.762 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T01:55:38.762 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T01:55:38.762 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T01:55:38.762 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T01:55:38.762 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T01:55:38.762 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T01:55:38.762 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T01:55:38.762 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T01:55:38.762 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T01:55:38.763 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T01:55:38.763 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T01:55:38.763 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T01:55:38.763 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T01:55:38.763 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T01:55:38.763 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T01:55:38.763 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T01:55:38.763 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T01:55:38.763 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T01:55:38.763 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T01:55:38.763 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T01:55:38.763 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T01:55:38.763 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T01:55:38.763 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T01:55:38.763 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T01:55:38.764 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T01:55:38.764 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T01:55:38.764 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T01:55:38.764 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T01:55:38.764 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T01:55:38.764 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T01:55:38.764 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T01:55:38.764 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T01:55:38.764 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T01:55:38.764 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T01:55:38.764 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T01:55:38.764 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T01:55:38.764 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T01:55:38.764 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T01:55:38.764 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.table_properties_collectors: 2024-03-26T01:55:38.764 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.inplace_update_support: 0 2024-03-26T01:55:38.765 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T01:55:38.765 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T01:55:38.765 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T01:55:38.765 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T01:55:38.765 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.bloom_locality: 0 2024-03-26T01:55:38.765 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.max_successive_merges: 0 2024-03-26T01:55:38.765 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T01:55:38.765 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T01:55:38.765 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T01:55:38.765 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T01:55:38.765 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.ttl: 2592000 2024-03-26T01:55:38.765 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T01:55:38.765 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T01:55:38.765 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T01:55:38.765 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.enable_blob_files: false 2024-03-26T01:55:38.766 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.min_blob_size: 0 2024-03-26T01:55:38.766 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T01:55:38.766 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T01:55:38.766 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T01:55:38.766 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T01:55:38.766 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T01:55:38.766 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T01:55:38.766 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T01:55:38.766 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T01:55:38.766 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-26T01:55:38.766 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-26T01:55:38.766 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2482f249-338d-4a5b-9b28-de10c954c8a9 2024-03-26T01:55:38.766 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418138705863, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-26T01:55:38.766 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-26T01:55:38.766 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418138706702, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 83951, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 266, "table_properties": {"data_size": 82130, "index_size": 231, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10814, "raw_average_key_size": 48, "raw_value_size": 76107, "raw_average_value_size": 338, "num_data_blocks": 10, "num_entries": 225, "num_filter_entries": 225, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711418138, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2482f249-338d-4a5b-9b28-de10c954c8a9", "db_session_id": "OKBH2VZV0R6CC88Y8LC3", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-26T01:55:38.767 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418138706785, "job": 1, "event": "recovery_finished"} 2024-03-26T01:55:38.767 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-26T01:55:38.767 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T01:55:38.767 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56265fa34000 2024-03-26T01:55:38.767 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: DB pointer 0x56265fa1e000 2024-03-26T01:55:38.767 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T01:55:38.767 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T01:55:38.767 INFO:journalctl@ceph.mon.a.smithi072.stdout: ** DB Stats ** 2024-03-26T01:55:38.767 INFO:journalctl@ceph.mon.a.smithi072.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T01:55:38.767 INFO:journalctl@ceph.mon.a.smithi072.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T01:55:38.767 INFO:journalctl@ceph.mon.a.smithi072.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T01:55:38.767 INFO:journalctl@ceph.mon.a.smithi072.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T01:55:38.767 INFO:journalctl@ceph.mon.a.smithi072.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T01:55:38.767 INFO:journalctl@ceph.mon.a.smithi072.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T01:55:38.767 INFO:journalctl@ceph.mon.a.smithi072.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T01:55:38.768 INFO:journalctl@ceph.mon.a.smithi072.stdout: 2024-03-26T01:55:38.768 INFO:journalctl@ceph.mon.a.smithi072.stdout: ** Compaction Stats [default] ** 2024-03-26T01:55:38.768 INFO:journalctl@ceph.mon.a.smithi072.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T01:55:38.768 INFO:journalctl@ceph.mon.a.smithi072.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T01:55:38.768 INFO:journalctl@ceph.mon.a.smithi072.stdout: L0 2/0 83.82 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T01:55:38.768 INFO:journalctl@ceph.mon.a.smithi072.stdout: Sum 2/0 83.82 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T01:55:38.768 INFO:journalctl@ceph.mon.a.smithi072.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T01:55:38.768 INFO:journalctl@ceph.mon.a.smithi072.stdout: 2024-03-26T01:55:38.768 INFO:journalctl@ceph.mon.a.smithi072.stdout: ** Compaction Stats [default] ** 2024-03-26T01:55:38.768 INFO:journalctl@ceph.mon.a.smithi072.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T01:55:38.768 INFO:journalctl@ceph.mon.a.smithi072.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T01:55:38.768 INFO:journalctl@ceph.mon.a.smithi072.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 150.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T01:55:38.769 INFO:journalctl@ceph.mon.a.smithi072.stdout: 2024-03-26T01:55:38.769 INFO:journalctl@ceph.mon.a.smithi072.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T01:55:38.769 INFO:journalctl@ceph.mon.a.smithi072.stdout: 2024-03-26T01:55:38.769 INFO:journalctl@ceph.mon.a.smithi072.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T01:55:38.769 INFO:journalctl@ceph.mon.a.smithi072.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T01:55:38.769 INFO:journalctl@ceph.mon.a.smithi072.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T01:55:38.769 INFO:journalctl@ceph.mon.a.smithi072.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T01:55:38.769 INFO:journalctl@ceph.mon.a.smithi072.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T01:55:38.769 INFO:journalctl@ceph.mon.a.smithi072.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T01:55:38.769 INFO:journalctl@ceph.mon.a.smithi072.stdout: Cumulative compaction: 0.00 GB write, 27.89 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T01:55:38.769 INFO:journalctl@ceph.mon.a.smithi072.stdout: Interval compaction: 0.00 GB write, 27.89 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T01:55:38.769 INFO:journalctl@ceph.mon.a.smithi072.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T01:55:38.769 INFO:journalctl@ceph.mon.a.smithi072.stdout: Block cache BinnedLRUCache@0x56265f943090#7 capacity: 512.00 MB usage: 1.19 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-26T01:55:38.769 INFO:journalctl@ceph.mon.a.smithi072.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.42 KB,8.04663e-05%) Misc(1,0.00 KB,0%) 2024-03-26T01:55:38.769 INFO:journalctl@ceph.mon.a.smithi072.stdout: 2024-03-26T01:55:38.770 INFO:journalctl@ceph.mon.a.smithi072.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T01:55:38.770 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: starting mon.a rank 0 at public addrs [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] at bind addrs [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T01:55:38.770 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: mon.a@-1(???) e1 preinit fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T01:55:38.770 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: mon.a@-1(???).mds e1 new map 2024-03-26T01:55:38.770 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: mon.a@-1(???).mds e1 print_map 2024-03-26T01:55:38.770 INFO:journalctl@ceph.mon.a.smithi072.stdout: e1 2024-03-26T01:55:38.770 INFO:journalctl@ceph.mon.a.smithi072.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T01:55:38.770 INFO:journalctl@ceph.mon.a.smithi072.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-26T01:55:38.770 INFO:journalctl@ceph.mon.a.smithi072.stdout: legacy client fscid: -1 2024-03-26T01:55:38.770 INFO:journalctl@ceph.mon.a.smithi072.stdout: 2024-03-26T01:55:38.770 INFO:journalctl@ceph.mon.a.smithi072.stdout: No filesystems configured 2024-03-26T01:55:38.770 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T01:55:38.770 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T01:55:38.770 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T01:55:38.770 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T01:55:38.771 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-26T01:55:38.771 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 bash[30139]: 52553733e232008acdda40c51484e26e7561b37ee6b2680a5a5090858c8021d4 2024-03-26T01:55:38.771 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 systemd[1]: Started Ceph mon.a for c86fbdd4-eb13-11ee-95d0-87774f69a715. 2024-03-26T01:55:38.771 INFO:teuthology.orchestra.run.smithi072.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-26T01:55:39.070 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T01:55:39.070 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: monmap e1: 1 mons at {a=[v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T01:55:39.070 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: fsmap 2024-03-26T01:55:39.070 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T01:55:39.070 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:38 smithi072 ceph-mon[30177]: mgrmap e1: no daemons active 2024-03-26T01:55:39.780 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.174+0000 7f0d50dfe700 1 Processor -- start 2024-03-26T01:55:39.785 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.174+0000 7f0d50dfe700 1 -- start start 2024-03-26T01:55:39.785 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.174+0000 7f0d50dfe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d4c1060f0 0x7f0d4c106510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:55:39.786 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.174+0000 7f0d50dfe700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c106ae0 con 0x7f0d4c1060f0 2024-03-26T01:55:39.786 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.175+0000 7f0d4a59c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d4c1060f0 0x7f0d4c106510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:55:39.786 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.175+0000 7f0d4a59c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d4c1060f0 0x7f0d4c106510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55750/0 (socket says 172.21.15.72:55750) 2024-03-26T01:55:39.786 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.175+0000 7f0d4a59c700 1 -- 172.21.15.72:0/3308308882 learned_addr learned my addr 172.21.15.72:0/3308308882 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:55:39.786 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.175+0000 7f0d4a59c700 1 -- 172.21.15.72:0/3308308882 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4c1072f0 con 0x7f0d4c1060f0 2024-03-26T01:55:39.786 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.175+0000 7f0d4a59c700 1 --2- 172.21.15.72:0/3308308882 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d4c1060f0 0x7f0d4c106510 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c0098a0 tx=0x7f0d3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27ca6f3c9770c91a server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:55:39.786 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.175+0000 7f0d4959a700 1 -- 172.21.15.72:0/3308308882 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d3c004160 con 0x7f0d4c1060f0 2024-03-26T01:55:39.786 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.175+0000 7f0d4959a700 1 -- 172.21.15.72:0/3308308882 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f0d3c0042c0 con 0x7f0d4c1060f0 2024-03-26T01:55:39.786 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.175+0000 7f0d4959a700 1 -- 172.21.15.72:0/3308308882 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d3c0119b0 con 0x7f0d4c1060f0 2024-03-26T01:55:39.786 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.176+0000 7f0d50dfe700 1 -- 172.21.15.72:0/3308308882 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d4c1060f0 msgr2=0x7f0d4c106510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:55:39.786 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.176+0000 7f0d50dfe700 1 --2- 172.21.15.72:0/3308308882 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d4c1060f0 0x7f0d4c106510 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c0098a0 tx=0x7f0d3c009bb0 comp rx=0 tx=0).stop 2024-03-26T01:55:39.786 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.176+0000 7f0d50dfe700 1 -- 172.21.15.72:0/3308308882 shutdown_connections 2024-03-26T01:55:39.787 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.176+0000 7f0d50dfe700 1 --2- 172.21.15.72:0/3308308882 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d4c1060f0 0x7f0d4c106510 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:39.787 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.176+0000 7f0d50dfe700 1 -- 172.21.15.72:0/3308308882 >> 172.21.15.72:0/3308308882 conn(0x7f0d4c1016b0 msgr2=0x7f0d4c103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:55:39.787 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.176+0000 7f0d50dfe700 1 -- 172.21.15.72:0/3308308882 shutdown_connections 2024-03-26T01:55:39.787 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.176+0000 7f0d50dfe700 1 -- 172.21.15.72:0/3308308882 wait complete. 2024-03-26T01:55:39.787 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.177+0000 7f0d50dfe700 1 Processor -- start 2024-03-26T01:55:39.787 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.177+0000 7f0d50dfe700 1 -- start start 2024-03-26T01:55:39.787 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.177+0000 7f0d50dfe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d4c1060f0 0x7f0d4c07bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:55:39.787 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.177+0000 7f0d50dfe700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d4c07c1b0 con 0x7f0d4c1060f0 2024-03-26T01:55:39.787 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.177+0000 7f0d4a59c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d4c1060f0 0x7f0d4c07bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:55:39.787 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.177+0000 7f0d4a59c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d4c1060f0 0x7f0d4c07bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55764/0 (socket says 172.21.15.72:55764) 2024-03-26T01:55:39.787 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.177+0000 7f0d4a59c700 1 -- 172.21.15.72:0/1319875681 learned_addr learned my addr 172.21.15.72:0/1319875681 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:55:39.787 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.177+0000 7f0d4a59c700 1 -- 172.21.15.72:0/1319875681 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d3c009550 con 0x7f0d4c1060f0 2024-03-26T01:55:39.787 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.177+0000 7f0d4a59c700 1 --2- 172.21.15.72:0/1319875681 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d4c1060f0 0x7f0d4c07bc70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c0046c0 tx=0x7f0d3c011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:55:39.787 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.178+0000 7f0d3b7fe700 1 -- 172.21.15.72:0/1319875681 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d3c0039c0 con 0x7f0d4c1060f0 2024-03-26T01:55:39.788 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.178+0000 7f0d50dfe700 1 -- 172.21.15.72:0/1319875681 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d4c07a3a0 con 0x7f0d4c1060f0 2024-03-26T01:55:39.788 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.178+0000 7f0d3b7fe700 1 -- 172.21.15.72:0/1319875681 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f0d3c003b20 con 0x7f0d4c1060f0 2024-03-26T01:55:39.788 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.178+0000 7f0d3b7fe700 1 -- 172.21.15.72:0/1319875681 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d3c01a780 con 0x7f0d4c1060f0 2024-03-26T01:55:39.788 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.178+0000 7f0d50dfe700 1 -- 172.21.15.72:0/1319875681 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d4c07a840 con 0x7f0d4c1060f0 2024-03-26T01:55:39.788 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.178+0000 7f0d3b7fe700 1 -- 172.21.15.72:0/1319875681 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f0d3c01a9f0 con 0x7f0d4c1060f0 2024-03-26T01:55:39.788 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.178+0000 7f0d3b7fe700 1 -- 172.21.15.72:0/1319875681 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0d3c02de40 con 0x7f0d4c1060f0 2024-03-26T01:55:39.788 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.179+0000 7f0d50dfe700 1 -- 172.21.15.72:0/1319875681 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d4c0636b0 con 0x7f0d4c1060f0 2024-03-26T01:55:39.788 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.181+0000 7f0d3b7fe700 1 -- 172.21.15.72:0/1319875681 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f0d3c024b50 con 0x7f0d4c1060f0 2024-03-26T01:55:39.788 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.247+0000 7f0d50dfe700 1 -- 172.21.15.72:0/1319875681 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f0d4c07ab40 con 0x7f0d4c1060f0 2024-03-26T01:55:39.788 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.251+0000 7f0d3b7fe700 1 -- 172.21.15.72:0/1319875681 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f0d3c024cf0 con 0x7f0d4c1060f0 2024-03-26T01:55:39.788 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.252+0000 7f0d50dfe700 1 -- 172.21.15.72:0/1319875681 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d4c1060f0 msgr2=0x7f0d4c07bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:55:39.788 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.252+0000 7f0d50dfe700 1 --2- 172.21.15.72:0/1319875681 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d4c1060f0 0x7f0d4c07bc70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c0046c0 tx=0x7f0d3c011a50 comp rx=0 tx=0).stop 2024-03-26T01:55:39.788 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.252+0000 7f0d50dfe700 1 -- 172.21.15.72:0/1319875681 shutdown_connections 2024-03-26T01:55:39.788 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.252+0000 7f0d50dfe700 1 --2- 172.21.15.72:0/1319875681 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0d4c1060f0 0x7f0d4c07bc70 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:39.789 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.252+0000 7f0d50dfe700 1 -- 172.21.15.72:0/1319875681 >> 172.21.15.72:0/1319875681 conn(0x7f0d4c1016b0 msgr2=0x7f0d4c1021a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:55:39.789 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.252+0000 7f0d50dfe700 1 -- 172.21.15.72:0/1319875681 shutdown_connections 2024-03-26T01:55:39.789 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:39.253+0000 7f0d50dfe700 1 -- 172.21.15.72:0/1319875681 wait complete. 2024-03-26T01:55:39.789 INFO:teuthology.orchestra.run.smithi072.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-26T01:55:39.811 INFO:teuthology.orchestra.run.smithi072.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-26T01:55:39.811 INFO:teuthology.orchestra.run.smithi072.stdout:Creating mgr... 2024-03-26T01:55:39.812 INFO:teuthology.orchestra.run.smithi072.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-26T01:55:39.812 INFO:teuthology.orchestra.run.smithi072.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-26T01:55:40.062 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@mgr.y 2024-03-26T01:55:40.062 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@mgr.y.service: Unit ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@mgr.y.service not loaded. 2024-03-26T01:55:40.202 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715.target.wants/ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@.service. 2024-03-26T01:55:40.541 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1319875681' entity='client.admin' 2024-03-26T01:55:40.541 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:40 smithi072 podman[30418]: 2024-03-26 01:55:40.34228963 +0000 UTC m=+0.017766383 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T01:55:40.752 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T01:55:40.752 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout disabled 2024-03-26T01:55:40.758 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T01:55:40.759 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout inactive 2024-03-26T01:55:40.759 INFO:teuthology.orchestra.run.smithi072.stdout:firewalld.service is not enabled 2024-03-26T01:55:40.759 INFO:teuthology.orchestra.run.smithi072.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T01:55:40.767 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T01:55:40.767 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout disabled 2024-03-26T01:55:40.774 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T01:55:40.774 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout inactive 2024-03-26T01:55:40.774 INFO:teuthology.orchestra.run.smithi072.stdout:firewalld.service is not enabled 2024-03-26T01:55:40.774 INFO:teuthology.orchestra.run.smithi072.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-26T01:55:40.774 INFO:teuthology.orchestra.run.smithi072.stdout:Waiting for mgr to start... 2024-03-26T01:55:40.774 INFO:teuthology.orchestra.run.smithi072.stdout:Waiting for mgr... 2024-03-26T01:55:40.903 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:40 smithi072 podman[30418]: 2024-03-26 01:55:40.638670291 +0000 UTC m=+0.314147033 container init fb1cfd7a9a912e8eb3a16927825a80cb19944ba977be10aec814ece8da829caa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-26T01:55:40.903 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:40 smithi072 podman[30418]: 2024-03-26 01:55:40.644079091 +0000 UTC m=+0.319555842 container start fb1cfd7a9a912e8eb3a16927825a80cb19944ba977be10aec814ece8da829caa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-26T01:55:40.903 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:40 smithi072 bash[30418]: fb1cfd7a9a912e8eb3a16927825a80cb19944ba977be10aec814ece8da829caa 2024-03-26T01:55:40.903 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:40 smithi072 systemd[1]: Started Ceph mgr.y for c86fbdd4-eb13-11ee-95d0-87774f69a715. 2024-03-26T01:55:41.459 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:41 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:41.196+0000 7fb1106cd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T01:55:41.459 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:41 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:41.293+0000 7fb1106cd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T01:55:41.718 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:41 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2317861653' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T01:55:42.021 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:41 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:41.995+0000 7fb1106cd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T01:55:42.024 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-03-26T01:55:42.027 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout { 2024-03-26T01:55:42.027 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "fsid": "c86fbdd4-eb13-11ee-95d0-87774f69a715", 2024-03-26T01:55:42.027 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T01:55:42.027 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T01:55:42.027 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T01:55:42.027 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T01:55:42.027 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:42.027 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T01:55:42.027 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T01:55:42.027 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 0 2024-03-26T01:55:42.028 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-03-26T01:55:42.028 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T01:55:42.028 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T01:55:42.028 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-03-26T01:55:42.028 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-26T01:55:42.028 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T01:55:42.028 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:42.028 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T01:55:42.028 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T01:55:42.028 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:42.028 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T01:55:42.028 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:42.028 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T01:55:42.028 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T01:55:42.028 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T01:55:42.036 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T01:55:42.036 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T01:55:42.036 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T01:55:42.036 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:42.036 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T01:55:42.037 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T01:55:42.037 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T01:55:42.037 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T01:55:42.037 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T01:55:42.037 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T01:55:42.037 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T01:55:42.037 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T01:55:42.037 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T01:55:42.037 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:42.037 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T01:55:42.037 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:42.037 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T01:55:42.037 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T01:55:42.038 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:42.038 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T01:55:42.038 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T01:55:42.038 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T01:55:42.038 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T01:55:42.038 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T01:55:42.038 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T01:55:42.038 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T01:55:42.038 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-03-26T01:55:42.038 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T01:55:42.038 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:42.038 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T01:55:42.038 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:42.038 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T01:55:31.587501+0000", 2024-03-26T01:55:42.039 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T01:55:42.039 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:42.039 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T01:55:42.039 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout } 2024-03-26T01:55:42.039 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.307+0000 7f23e303f700 1 Processor -- start 2024-03-26T01:55:42.039 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.307+0000 7f23e303f700 1 -- start start 2024-03-26T01:55:42.039 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.307+0000 7f23e303f700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23dc1049b0 0x7f23dc104dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:55:42.039 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.307+0000 7f23e303f700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23dc0796a0 con 0x7f23dc1049b0 2024-03-26T01:55:42.039 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.308+0000 7f23e0ddb700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23dc1049b0 0x7f23dc104dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:55:42.039 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.308+0000 7f23e0ddb700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23dc1049b0 0x7f23dc104dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55780/0 (socket says 172.21.15.72:55780) 2024-03-26T01:55:42.039 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.308+0000 7f23e0ddb700 1 -- 172.21.15.72:0/2808774294 learned_addr learned my addr 172.21.15.72:0/2808774294 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:55:42.039 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.308+0000 7f23e0ddb700 1 -- 172.21.15.72:0/2808774294 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23dc105310 con 0x7f23dc1049b0 2024-03-26T01:55:42.039 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.308+0000 7f23e0ddb700 1 --2- 172.21.15.72:0/2808774294 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23dc1049b0 0x7f23dc104dd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f23d00098a0 tx=0x7f23d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec2c89538f293107 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:55:42.039 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.308+0000 7f23db7fe700 1 -- 172.21.15.72:0/2808774294 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f23d0004160 con 0x7f23dc1049b0 2024-03-26T01:55:42.040 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.308+0000 7f23db7fe700 1 -- 172.21.15.72:0/2808774294 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f23d00042c0 con 0x7f23dc1049b0 2024-03-26T01:55:42.040 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.308+0000 7f23db7fe700 1 -- 172.21.15.72:0/2808774294 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f23d00119b0 con 0x7f23dc1049b0 2024-03-26T01:55:42.040 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.309+0000 7f23e303f700 1 -- 172.21.15.72:0/2808774294 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23dc1049b0 msgr2=0x7f23dc104dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:55:42.040 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.309+0000 7f23e303f700 1 --2- 172.21.15.72:0/2808774294 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23dc1049b0 0x7f23dc104dd0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f23d00098a0 tx=0x7f23d0009bb0 comp rx=0 tx=0).stop 2024-03-26T01:55:42.040 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.309+0000 7f23e303f700 1 -- 172.21.15.72:0/2808774294 shutdown_connections 2024-03-26T01:55:42.040 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.309+0000 7f23e303f700 1 --2- 172.21.15.72:0/2808774294 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23dc1049b0 0x7f23dc104dd0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:42.040 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.309+0000 7f23e303f700 1 -- 172.21.15.72:0/2808774294 >> 172.21.15.72:0/2808774294 conn(0x7f23dc100590 msgr2=0x7f23dc1029b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:55:42.040 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.309+0000 7f23e303f700 1 -- 172.21.15.72:0/2808774294 shutdown_connections 2024-03-26T01:55:42.040 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.309+0000 7f23e303f700 1 -- 172.21.15.72:0/2808774294 wait complete. 2024-03-26T01:55:42.040 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.310+0000 7f23e303f700 1 Processor -- start 2024-03-26T01:55:42.040 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.310+0000 7f23e303f700 1 -- start start 2024-03-26T01:55:42.040 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.310+0000 7f23e303f700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23dc1049b0 0x7f23dc19c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:55:42.040 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.310+0000 7f23e303f700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23dc19c580 con 0x7f23dc1049b0 2024-03-26T01:55:42.040 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.310+0000 7f23e0ddb700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23dc1049b0 0x7f23dc19c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:55:42.041 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.310+0000 7f23e0ddb700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23dc1049b0 0x7f23dc19c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:55786/0 (socket says 172.21.15.72:55786) 2024-03-26T01:55:42.041 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.310+0000 7f23e0ddb700 1 -- 172.21.15.72:0/2317861653 learned_addr learned my addr 172.21.15.72:0/2317861653 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:55:42.041 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.310+0000 7f23e0ddb700 1 -- 172.21.15.72:0/2317861653 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d0009550 con 0x7f23dc1049b0 2024-03-26T01:55:42.041 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.311+0000 7f23e0ddb700 1 --2- 172.21.15.72:0/2317861653 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23dc1049b0 0x7f23dc19c040 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f23d0009130 tx=0x7f23d0011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:55:42.041 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.311+0000 7f23d9ffb700 1 -- 172.21.15.72:0/2317861653 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f23d0003aa0 con 0x7f23dc1049b0 2024-03-26T01:55:42.041 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.311+0000 7f23d9ffb700 1 -- 172.21.15.72:0/2317861653 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f23d001a8c0 con 0x7f23dc1049b0 2024-03-26T01:55:42.041 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.311+0000 7f23e303f700 1 -- 172.21.15.72:0/2317861653 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23dc19c780 con 0x7f23dc1049b0 2024-03-26T01:55:42.041 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.311+0000 7f23d9ffb700 1 -- 172.21.15.72:0/2317861653 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f23d0019790 con 0x7f23dc1049b0 2024-03-26T01:55:42.041 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.311+0000 7f23e303f700 1 -- 172.21.15.72:0/2317861653 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23dc19cc20 con 0x7f23dc1049b0 2024-03-26T01:55:42.041 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.311+0000 7f23d9ffb700 1 -- 172.21.15.72:0/2317861653 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f23d0025460 con 0x7f23dc1049b0 2024-03-26T01:55:42.041 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.311+0000 7f23d9ffb700 1 -- 172.21.15.72:0/2317861653 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f23d000bb20 con 0x7f23dc1049b0 2024-03-26T01:55:42.041 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.312+0000 7f23e303f700 1 -- 172.21.15.72:0/2317861653 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23dc195c90 con 0x7f23dc1049b0 2024-03-26T01:55:42.041 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.314+0000 7f23d9ffb700 1 -- 172.21.15.72:0/2317861653 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f23d002a060 con 0x7f23dc1049b0 2024-03-26T01:55:42.042 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.442+0000 7f23e303f700 1 -- 172.21.15.72:0/2317861653 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f23dc106390 con 0x7f23dc1049b0 2024-03-26T01:55:42.042 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.443+0000 7f23d9ffb700 1 -- 172.21.15.72:0/2317861653 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f23d001aa30 con 0x7f23dc1049b0 2024-03-26T01:55:42.042 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.444+0000 7f23e303f700 1 -- 172.21.15.72:0/2317861653 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23dc1049b0 msgr2=0x7f23dc19c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:55:42.042 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.444+0000 7f23e303f700 1 --2- 172.21.15.72:0/2317861653 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23dc1049b0 0x7f23dc19c040 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f23d0009130 tx=0x7f23d0011a50 comp rx=0 tx=0).stop 2024-03-26T01:55:42.042 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.444+0000 7f23e303f700 1 -- 172.21.15.72:0/2317861653 shutdown_connections 2024-03-26T01:55:42.042 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.444+0000 7f23e303f700 1 --2- 172.21.15.72:0/2317861653 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f23dc1049b0 0x7f23dc19c040 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:42.042 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.444+0000 7f23e303f700 1 -- 172.21.15.72:0/2317861653 >> 172.21.15.72:0/2317861653 conn(0x7f23dc100590 msgr2=0x7f23dc192dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:55:42.042 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.444+0000 7f23e303f700 1 -- 172.21.15.72:0/2317861653 shutdown_connections 2024-03-26T01:55:42.042 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:41.444+0000 7f23e303f700 1 -- 172.21.15.72:0/2317861653 wait complete. 2024-03-26T01:55:42.042 INFO:teuthology.orchestra.run.smithi072.stdout:mgr not available, waiting (1/15)... 2024-03-26T01:55:42.327 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:42 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:42.092+0000 7fb1106cd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T01:55:42.327 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:42 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:42.232+0000 7fb1106cd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T01:55:42.327 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:42 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:42.326+0000 7fb1106cd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T01:55:42.665 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:42 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:42.407+0000 7fb1106cd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T01:55:42.665 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:42 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:42.664+0000 7fb1106cd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T01:55:42.949 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:42 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:42.771+0000 7fb1106cd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T01:55:42.949 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:42 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:42.856+0000 7fb1106cd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T01:55:42.949 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:42 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:42.949+0000 7fb1106cd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T01:55:43.218 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:43 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:43.121+0000 7fb1106cd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T01:55:44.410 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:44.080+0000 7fb1106cd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T01:55:44.410 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:44.210+0000 7fb1106cd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T01:55:44.663 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:44.567+0000 7fb1106cd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T01:55:44.663 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:44.659+0000 7fb1106cd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T01:55:44.959 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:44 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2526080463' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T01:55:44.959 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:44.854+0000 7fb1106cd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T01:55:45.218 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:45 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:45.040+0000 7fb1106cd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T01:55:45.224 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-03-26T01:55:45.224 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout { 2024-03-26T01:55:45.224 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "fsid": "c86fbdd4-eb13-11ee-95d0-87774f69a715", 2024-03-26T01:55:45.224 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T01:55:45.224 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T01:55:45.224 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T01:55:45.225 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T01:55:45.225 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:45.226 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T01:55:45.226 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T01:55:45.226 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 0 2024-03-26T01:55:45.226 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-03-26T01:55:45.226 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T01:55:45.226 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T01:55:45.226 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-03-26T01:55:45.226 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-26T01:55:45.226 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T01:55:45.226 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:45.226 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T01:55:45.227 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T01:55:45.227 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:45.227 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T01:55:45.227 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:45.227 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T01:55:45.227 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T01:55:45.227 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T01:55:45.235 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T01:55:45.235 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T01:55:45.236 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T01:55:45.236 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:45.236 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T01:55:45.236 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T01:55:45.236 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T01:55:45.236 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T01:55:45.236 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T01:55:45.236 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T01:55:45.237 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T01:55:45.237 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T01:55:45.237 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T01:55:45.237 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:45.237 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T01:55:45.237 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:45.237 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T01:55:45.237 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T01:55:45.237 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:45.237 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T01:55:45.237 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T01:55:45.237 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T01:55:45.238 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T01:55:45.238 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T01:55:45.238 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T01:55:45.238 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T01:55:45.238 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-03-26T01:55:45.238 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T01:55:45.238 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:45.238 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T01:55:45.238 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:45.238 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T01:55:31.587501+0000", 2024-03-26T01:55:45.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T01:55:45.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:45.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T01:55:45.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout } 2024-03-26T01:55:45.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.512+0000 7f5aa867d700 1 Processor -- start 2024-03-26T01:55:45.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.512+0000 7f5aa867d700 1 -- start start 2024-03-26T01:55:45.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.513+0000 7f5aa867d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5aa01049d0 0x7f5aa0104df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:55:45.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.513+0000 7f5aa867d700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa0079f50 con 0x7f5aa01049d0 2024-03-26T01:55:45.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.513+0000 7f5aa6419700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5aa01049d0 0x7f5aa0104df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:55:45.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.513+0000 7f5aa6419700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5aa01049d0 0x7f5aa0104df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44304/0 (socket says 172.21.15.72:44304) 2024-03-26T01:55:45.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.513+0000 7f5aa6419700 1 -- 172.21.15.72:0/3756789255 learned_addr learned my addr 172.21.15.72:0/3756789255 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:55:45.239 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.513+0000 7f5aa6419700 1 -- 172.21.15.72:0/3756789255 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa0105330 con 0x7f5aa01049d0 2024-03-26T01:55:45.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.513+0000 7f5aa6419700 1 --2- 172.21.15.72:0/3756789255 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5aa01049d0 0x7f5aa0104df0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c0098a0 tx=0x7f5a9c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d8d0c9f983c9ac0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:55:45.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.513+0000 7f5aa5417700 1 -- 172.21.15.72:0/3756789255 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5a9c004160 con 0x7f5aa01049d0 2024-03-26T01:55:45.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.514+0000 7f5aa5417700 1 -- 172.21.15.72:0/3756789255 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f5a9c0042c0 con 0x7f5aa01049d0 2024-03-26T01:55:45.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.514+0000 7f5aa5417700 1 -- 172.21.15.72:0/3756789255 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5a9c0119b0 con 0x7f5aa01049d0 2024-03-26T01:55:45.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.514+0000 7f5aa867d700 1 -- 172.21.15.72:0/3756789255 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5aa01049d0 msgr2=0x7f5aa0104df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:55:45.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.514+0000 7f5aa867d700 1 --2- 172.21.15.72:0/3756789255 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5aa01049d0 0x7f5aa0104df0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c0098a0 tx=0x7f5a9c009bb0 comp rx=0 tx=0).stop 2024-03-26T01:55:45.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.514+0000 7f5aa867d700 1 -- 172.21.15.72:0/3756789255 shutdown_connections 2024-03-26T01:55:45.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.514+0000 7f5aa867d700 1 --2- 172.21.15.72:0/3756789255 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5aa01049d0 0x7f5aa0104df0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:45.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.514+0000 7f5aa867d700 1 -- 172.21.15.72:0/3756789255 >> 172.21.15.72:0/3756789255 conn(0x7f5aa01005d0 msgr2=0x7f5aa01029f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:55:45.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.514+0000 7f5aa867d700 1 -- 172.21.15.72:0/3756789255 shutdown_connections 2024-03-26T01:55:45.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.514+0000 7f5aa867d700 1 -- 172.21.15.72:0/3756789255 wait complete. 2024-03-26T01:55:45.240 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.515+0000 7f5aa867d700 1 Processor -- start 2024-03-26T01:55:45.241 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.515+0000 7f5aa867d700 1 -- start start 2024-03-26T01:55:45.241 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.515+0000 7f5aa867d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5aa01049d0 0x7f5aa019c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:55:45.241 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.515+0000 7f5aa867d700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa019c580 con 0x7f5aa01049d0 2024-03-26T01:55:45.241 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.515+0000 7f5aa6419700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5aa01049d0 0x7f5aa019c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:55:45.241 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.515+0000 7f5aa6419700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5aa01049d0 0x7f5aa019c040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44314/0 (socket says 172.21.15.72:44314) 2024-03-26T01:55:45.241 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.515+0000 7f5aa6419700 1 -- 172.21.15.72:0/2526080463 learned_addr learned my addr 172.21.15.72:0/2526080463 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:55:45.241 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.515+0000 7f5aa6419700 1 -- 172.21.15.72:0/2526080463 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a9c009550 con 0x7f5aa01049d0 2024-03-26T01:55:45.241 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.515+0000 7f5aa6419700 1 --2- 172.21.15.72:0/2526080463 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5aa01049d0 0x7f5aa019c040 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c0046c0 tx=0x7f5a9c011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:55:45.241 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.516+0000 7f5a977fe700 1 -- 172.21.15.72:0/2526080463 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5a9c0039c0 con 0x7f5aa01049d0 2024-03-26T01:55:45.241 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.516+0000 7f5aa867d700 1 -- 172.21.15.72:0/2526080463 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aa019c780 con 0x7f5aa01049d0 2024-03-26T01:55:45.241 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.516+0000 7f5a977fe700 1 -- 172.21.15.72:0/2526080463 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f5a9c003b20 con 0x7f5aa01049d0 2024-03-26T01:55:45.241 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.516+0000 7f5a977fe700 1 -- 172.21.15.72:0/2526080463 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5a9c01a7b0 con 0x7f5aa01049d0 2024-03-26T01:55:45.242 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.516+0000 7f5aa867d700 1 -- 172.21.15.72:0/2526080463 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aa019cc20 con 0x7f5aa01049d0 2024-03-26T01:55:45.242 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.516+0000 7f5a977fe700 1 -- 172.21.15.72:0/2526080463 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f5a9c01aa20 con 0x7f5aa01049d0 2024-03-26T01:55:45.242 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.516+0000 7f5a977fe700 1 -- 172.21.15.72:0/2526080463 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5a9c019700 con 0x7f5aa01049d0 2024-03-26T01:55:45.242 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.517+0000 7f5aa867d700 1 -- 172.21.15.72:0/2526080463 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aa0195cb0 con 0x7f5aa01049d0 2024-03-26T01:55:45.242 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.519+0000 7f5a977fe700 1 -- 172.21.15.72:0/2526080463 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f5a9c02de40 con 0x7f5aa01049d0 2024-03-26T01:55:45.242 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.611+0000 7f5aa867d700 1 -- 172.21.15.72:0/2526080463 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5aa01063b0 con 0x7f5aa01049d0 2024-03-26T01:55:45.242 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.612+0000 7f5a977fe700 1 -- 172.21.15.72:0/2526080463 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f5a9c024b30 con 0x7f5aa01049d0 2024-03-26T01:55:45.242 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.613+0000 7f5aa867d700 1 -- 172.21.15.72:0/2526080463 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5aa01049d0 msgr2=0x7f5aa019c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:55:45.242 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.613+0000 7f5aa867d700 1 --2- 172.21.15.72:0/2526080463 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5aa01049d0 0x7f5aa019c040 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c0046c0 tx=0x7f5a9c011a50 comp rx=0 tx=0).stop 2024-03-26T01:55:45.242 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.613+0000 7f5aa867d700 1 -- 172.21.15.72:0/2526080463 shutdown_connections 2024-03-26T01:55:45.242 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.613+0000 7f5aa867d700 1 --2- 172.21.15.72:0/2526080463 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5aa01049d0 0x7f5aa019c040 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:45.243 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.613+0000 7f5aa867d700 1 -- 172.21.15.72:0/2526080463 >> 172.21.15.72:0/2526080463 conn(0x7f5aa01005d0 msgr2=0x7f5aa0192df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:55:45.243 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.613+0000 7f5aa867d700 1 -- 172.21.15.72:0/2526080463 shutdown_connections 2024-03-26T01:55:45.243 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:44.613+0000 7f5aa867d700 1 -- 172.21.15.72:0/2526080463 wait complete. 2024-03-26T01:55:45.243 INFO:teuthology.orchestra.run.smithi072.stdout:mgr not available, waiting (2/15)... 2024-03-26T01:55:45.577 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:45 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:45.322+0000 7fb1106cd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T01:55:45.968 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:45 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:45.576+0000 7fb1106cd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T01:55:45.968 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:45 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:45.810+0000 7fb1106cd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T01:55:46.968 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:46 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:46.600+0000 7fb1106cd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T01:55:47.465 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:47 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:47.165+0000 7fb1106cd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T01:55:47.465 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:47 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:47.268+0000 7fb1106cd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T01:55:47.718 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:47 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:47.490+0000 7fb1106cd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T01:55:47.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:47 smithi072 ceph-mon[30177]: Activating manager daemon y 2024-03-26T01:55:47.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:47 smithi072 ceph-mon[30177]: mgrmap e2: y(active, starting, since 0.0031036s) 2024-03-26T01:55:47.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:47 smithi072 ceph-mon[30177]: from='mgr.14100 172.21.15.72:0/293053748' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T01:55:47.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:47 smithi072 ceph-mon[30177]: from='mgr.14100 172.21.15.72:0/293053748' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T01:55:47.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:47 smithi072 ceph-mon[30177]: from='mgr.14100 172.21.15.72:0/293053748' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T01:55:47.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:47 smithi072 ceph-mon[30177]: from='mgr.14100 172.21.15.72:0/293053748' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:55:47.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:47 smithi072 ceph-mon[30177]: from='mgr.14100 172.21.15.72:0/293053748' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T01:55:47.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:47 smithi072 ceph-mon[30177]: Manager daemon y is now available 2024-03-26T01:55:47.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:47 smithi072 ceph-mon[30177]: from='mgr.14100 172.21.15.72:0/293053748' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:55:47.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:47 smithi072 ceph-mon[30177]: from='mgr.14100 172.21.15.72:0/293053748' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T01:55:47.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:47 smithi072 ceph-mon[30177]: from='mgr.14100 172.21.15.72:0/293053748' entity='mgr.y' 2024-03-26T01:55:47.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:47 smithi072 ceph-mon[30177]: from='mgr.14100 172.21.15.72:0/293053748' entity='mgr.y' 2024-03-26T01:55:47.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:47 smithi072 ceph-mon[30177]: from='mgr.14100 172.21.15.72:0/293053748' entity='mgr.y' 2024-03-26T01:55:48.382 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-03-26T01:55:48.382 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout { 2024-03-26T01:55:48.382 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "fsid": "c86fbdd4-eb13-11ee-95d0-87774f69a715", 2024-03-26T01:55:48.382 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T01:55:48.382 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T01:55:48.382 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T01:55:48.382 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T01:55:48.383 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:48.383 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T01:55:48.383 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T01:55:48.383 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 0 2024-03-26T01:55:48.383 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-03-26T01:55:48.383 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T01:55:48.383 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T01:55:48.383 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-03-26T01:55:48.383 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-26T01:55:48.383 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T01:55:48.383 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:48.383 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T01:55:48.383 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T01:55:48.383 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:48.384 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T01:55:48.384 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:48.384 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T01:55:48.384 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T01:55:48.384 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T01:55:48.391 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T01:55:48.392 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T01:55:48.392 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T01:55:48.392 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:48.392 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T01:55:48.392 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T01:55:48.392 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T01:55:48.392 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T01:55:48.392 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T01:55:48.392 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T01:55:48.392 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T01:55:48.392 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T01:55:48.392 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T01:55:48.392 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:48.392 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T01:55:48.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:48.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T01:55:48.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T01:55:48.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:48.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T01:55:48.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T01:55:48.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T01:55:48.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T01:55:48.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T01:55:48.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T01:55:48.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T01:55:48.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-03-26T01:55:48.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T01:55:48.393 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:48.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T01:55:48.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:48.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T01:55:31.587501+0000", 2024-03-26T01:55:48.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T01:55:48.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:48.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T01:55:48.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout } 2024-03-26T01:55:48.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.724+0000 7f4db0787700 1 Processor -- start 2024-03-26T01:55:48.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.724+0000 7f4db0787700 1 -- start start 2024-03-26T01:55:48.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.724+0000 7f4db0787700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4da807bfe0 0x7f4da807a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:55:48.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.724+0000 7f4db0787700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da807aa10 con 0x7f4da807bfe0 2024-03-26T01:55:48.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.724+0000 7f4dae523700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4da807bfe0 0x7f4da807a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:55:48.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.724+0000 7f4dae523700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4da807bfe0 0x7f4da807a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44414/0 (socket says 172.21.15.72:44414) 2024-03-26T01:55:48.394 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.725+0000 7f4dae523700 1 -- 172.21.15.72:0/486120069 learned_addr learned my addr 172.21.15.72:0/486120069 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:55:48.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.725+0000 7f4dae523700 1 -- 172.21.15.72:0/486120069 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da807ab50 con 0x7f4da807bfe0 2024-03-26T01:55:48.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.725+0000 7f4dae523700 1 --2- 172.21.15.72:0/486120069 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4da807bfe0 0x7f4da807a440 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4da40098a0 tx=0x7f4da4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6d4198de266dc5e server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:55:48.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.725+0000 7f4dad521700 1 -- 172.21.15.72:0/486120069 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4da4004160 con 0x7f4da807bfe0 2024-03-26T01:55:48.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.725+0000 7f4dad521700 1 -- 172.21.15.72:0/486120069 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f4da40042c0 con 0x7f4da807bfe0 2024-03-26T01:55:48.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.725+0000 7f4dad521700 1 -- 172.21.15.72:0/486120069 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4da40119b0 con 0x7f4da807bfe0 2024-03-26T01:55:48.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.726+0000 7f4db0787700 1 -- 172.21.15.72:0/486120069 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4da807bfe0 msgr2=0x7f4da807a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:55:48.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.726+0000 7f4db0787700 1 --2- 172.21.15.72:0/486120069 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4da807bfe0 0x7f4da807a440 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4da40098a0 tx=0x7f4da4009bb0 comp rx=0 tx=0).stop 2024-03-26T01:55:48.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.726+0000 7f4db0787700 1 -- 172.21.15.72:0/486120069 shutdown_connections 2024-03-26T01:55:48.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.726+0000 7f4db0787700 1 --2- 172.21.15.72:0/486120069 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4da807bfe0 0x7f4da807a440 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:48.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.726+0000 7f4db0787700 1 -- 172.21.15.72:0/486120069 >> 172.21.15.72:0/486120069 conn(0x7f4da81016b0 msgr2=0x7f4da8103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:55:48.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.726+0000 7f4db0787700 1 -- 172.21.15.72:0/486120069 shutdown_connections 2024-03-26T01:55:48.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.726+0000 7f4db0787700 1 -- 172.21.15.72:0/486120069 wait complete. 2024-03-26T01:55:48.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.726+0000 7f4db0787700 1 Processor -- start 2024-03-26T01:55:48.395 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.726+0000 7f4db0787700 1 -- start start 2024-03-26T01:55:48.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.727+0000 7f4db0787700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4da807bfe0 0x7f4da81a0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:55:48.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.727+0000 7f4db0787700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da81a09b0 con 0x7f4da807bfe0 2024-03-26T01:55:48.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.727+0000 7f4dae523700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4da807bfe0 0x7f4da81a0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:55:48.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.727+0000 7f4dae523700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4da807bfe0 0x7f4da81a0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44418/0 (socket says 172.21.15.72:44418) 2024-03-26T01:55:48.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.727+0000 7f4dae523700 1 -- 172.21.15.72:0/3975005431 learned_addr learned my addr 172.21.15.72:0/3975005431 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:55:48.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.727+0000 7f4dae523700 1 -- 172.21.15.72:0/3975005431 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da4009550 con 0x7f4da807bfe0 2024-03-26T01:55:48.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.727+0000 7f4dae523700 1 --2- 172.21.15.72:0/3975005431 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4da807bfe0 0x7f4da81a0470 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4da40046c0 tx=0x7f4da4011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:55:48.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.727+0000 7f4d9f7fe700 1 -- 172.21.15.72:0/3975005431 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4da40039c0 con 0x7f4da807bfe0 2024-03-26T01:55:48.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.727+0000 7f4d9f7fe700 1 -- 172.21.15.72:0/3975005431 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f4da4003b20 con 0x7f4da807bfe0 2024-03-26T01:55:48.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.727+0000 7f4db0787700 1 -- 172.21.15.72:0/3975005431 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4da81a0bb0 con 0x7f4da807bfe0 2024-03-26T01:55:48.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.727+0000 7f4d9f7fe700 1 -- 172.21.15.72:0/3975005431 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4da401a780 con 0x7f4da807bfe0 2024-03-26T01:55:48.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.727+0000 7f4db0787700 1 -- 172.21.15.72:0/3975005431 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4da81a1050 con 0x7f4da807bfe0 2024-03-26T01:55:48.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.728+0000 7f4d9f7fe700 1 -- 172.21.15.72:0/3975005431 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 2) v1 ==== 45684+0+0 (secure 0 0 0) 0x7f4da4024430 con 0x7f4da807bfe0 2024-03-26T01:55:48.396 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.728+0000 7f4db0787700 1 -- 172.21.15.72:0/3975005431 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4da8199b90 con 0x7f4da807bfe0 2024-03-26T01:55:48.397 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.728+0000 7f4d9f7fe700 1 -- 172.21.15.72:0/3975005431 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4da404cc40 con 0x7f4da807bfe0 2024-03-26T01:55:48.397 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.730+0000 7f4d9f7fe700 1 -- 172.21.15.72:0/3975005431 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f4da40246e0 con 0x7f4da807bfe0 2024-03-26T01:55:48.397 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.800+0000 7f4db0787700 1 -- 172.21.15.72:0/3975005431 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4da81080c0 con 0x7f4da807bfe0 2024-03-26T01:55:48.397 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.801+0000 7f4d9f7fe700 1 -- 172.21.15.72:0/3975005431 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f4da4041e60 con 0x7f4da807bfe0 2024-03-26T01:55:48.397 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.801+0000 7f4db0787700 1 -- 172.21.15.72:0/3975005431 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4da807bfe0 msgr2=0x7f4da81a0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:55:48.397 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.801+0000 7f4db0787700 1 --2- 172.21.15.72:0/3975005431 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4da807bfe0 0x7f4da81a0470 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4da40046c0 tx=0x7f4da4011a50 comp rx=0 tx=0).stop 2024-03-26T01:55:48.397 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.802+0000 7f4db0787700 1 -- 172.21.15.72:0/3975005431 shutdown_connections 2024-03-26T01:55:48.397 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.802+0000 7f4db0787700 1 --2- 172.21.15.72:0/3975005431 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4da807bfe0 0x7f4da81a0470 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:48.397 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.802+0000 7f4db0787700 1 -- 172.21.15.72:0/3975005431 >> 172.21.15.72:0/3975005431 conn(0x7f4da81016b0 msgr2=0x7f4da8103aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:55:48.397 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.802+0000 7f4db0787700 1 -- 172.21.15.72:0/3975005431 shutdown_connections 2024-03-26T01:55:48.397 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:47.802+0000 7f4db0787700 1 -- 172.21.15.72:0/3975005431 wait complete. 2024-03-26T01:55:48.397 INFO:teuthology.orchestra.run.smithi072.stdout:mgr not available, waiting (3/15)... 2024-03-26T01:55:48.718 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:48 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3975005431' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T01:55:48.718 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:48 smithi072 ceph-mon[30177]: mgrmap e3: y(active, since 1.00634s) 2024-03-26T01:55:50.754 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:50 smithi072 ceph-mon[30177]: mgrmap e4: y(active, since 2s) 2024-03-26T01:55:51.718 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:51 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2497997394' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T01:55:51.729 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-03-26T01:55:51.729 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout { 2024-03-26T01:55:51.729 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "fsid": "c86fbdd4-eb13-11ee-95d0-87774f69a715", 2024-03-26T01:55:51.729 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T01:55:51.729 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T01:55:51.729 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T01:55:51.730 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T01:55:51.730 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:51.730 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T01:55:51.730 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T01:55:51.730 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 0 2024-03-26T01:55:51.730 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-03-26T01:55:51.730 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T01:55:51.730 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T01:55:51.730 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-03-26T01:55:51.730 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-26T01:55:51.730 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T01:55:51.731 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:51.731 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T01:55:51.731 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T01:55:51.731 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:51.731 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T01:55:51.731 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:51.731 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T01:55:51.731 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T01:55:51.740 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T01:55:51.741 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T01:55:51.741 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T01:55:51.741 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T01:55:51.741 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:51.741 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T01:55:51.741 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T01:55:51.741 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T01:55:51.741 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T01:55:51.741 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T01:55:51.741 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T01:55:51.741 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T01:55:51.742 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T01:55:51.742 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T01:55:51.742 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:51.742 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T01:55:51.742 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:51.742 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T01:55:51.742 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T01:55:51.742 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:51.742 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T01:55:51.742 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T01:55:51.742 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T01:55:51.742 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T01:55:51.743 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T01:55:51.743 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T01:55:51.743 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T01:55:51.743 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ], 2024-03-26T01:55:51.743 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T01:55:51.743 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:51.743 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T01:55:51.743 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T01:55:51.743 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T01:55:31.587501+0000", 2024-03-26T01:55:51.743 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T01:55:51.743 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout }, 2024-03-26T01:55:51.743 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T01:55:51.744 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout } 2024-03-26T01:55:51.744 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.851+0000 7f8e7e18c700 1 Processor -- start 2024-03-26T01:55:51.744 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.851+0000 7f8e7e18c700 1 -- start start 2024-03-26T01:55:51.744 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.852+0000 7f8e7e18c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e78104430 0x7f8e78104850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:55:51.744 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.852+0000 7f8e7e18c700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e78104e20 con 0x7f8e78104430 2024-03-26T01:55:51.744 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.852+0000 7f8e777fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e78104430 0x7f8e78104850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:55:51.744 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.852+0000 7f8e777fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e78104430 0x7f8e78104850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44430/0 (socket says 172.21.15.72:44430) 2024-03-26T01:55:51.744 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.852+0000 7f8e777fe700 1 -- 172.21.15.72:0/3920576172 learned_addr learned my addr 172.21.15.72:0/3920576172 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:55:51.744 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.852+0000 7f8e777fe700 1 -- 172.21.15.72:0/3920576172 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e781073d0 con 0x7f8e78104430 2024-03-26T01:55:51.744 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.852+0000 7f8e777fe700 1 --2- 172.21.15.72:0/3920576172 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e78104430 0x7f8e78104850 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c0098a0 tx=0x7f8e6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=925e85d22dcf8662 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:55:51.745 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.852+0000 7f8e767fc700 1 -- 172.21.15.72:0/3920576172 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e6c004160 con 0x7f8e78104430 2024-03-26T01:55:51.745 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.852+0000 7f8e767fc700 1 -- 172.21.15.72:0/3920576172 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f8e6c0042c0 con 0x7f8e78104430 2024-03-26T01:55:51.745 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.852+0000 7f8e767fc700 1 -- 172.21.15.72:0/3920576172 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e6c0119b0 con 0x7f8e78104430 2024-03-26T01:55:51.745 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.853+0000 7f8e7e18c700 1 -- 172.21.15.72:0/3920576172 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e78104430 msgr2=0x7f8e78104850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:55:51.745 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.853+0000 7f8e7e18c700 1 --2- 172.21.15.72:0/3920576172 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e78104430 0x7f8e78104850 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c0098a0 tx=0x7f8e6c009bb0 comp rx=0 tx=0).stop 2024-03-26T01:55:51.745 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.853+0000 7f8e7e18c700 1 -- 172.21.15.72:0/3920576172 shutdown_connections 2024-03-26T01:55:51.745 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.853+0000 7f8e7e18c700 1 --2- 172.21.15.72:0/3920576172 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e78104430 0x7f8e78104850 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:51.745 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.853+0000 7f8e7e18c700 1 -- 172.21.15.72:0/3920576172 >> 172.21.15.72:0/3920576172 conn(0x7f8e780ffd10 msgr2=0x7f8e78102130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:55:51.745 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.853+0000 7f8e7e18c700 1 -- 172.21.15.72:0/3920576172 shutdown_connections 2024-03-26T01:55:51.745 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.853+0000 7f8e7e18c700 1 -- 172.21.15.72:0/3920576172 wait complete. 2024-03-26T01:55:51.745 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.854+0000 7f8e7e18c700 1 Processor -- start 2024-03-26T01:55:51.745 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.854+0000 7f8e7e18c700 1 -- start start 2024-03-26T01:55:51.746 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.854+0000 7f8e7e18c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e78104430 0x7f8e78199e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:55:51.746 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.854+0000 7f8e7e18c700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7819a390 con 0x7f8e78104430 2024-03-26T01:55:51.746 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.854+0000 7f8e777fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e78104430 0x7f8e78199e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:55:51.746 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.854+0000 7f8e777fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e78104430 0x7f8e78199e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44444/0 (socket says 172.21.15.72:44444) 2024-03-26T01:55:51.746 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.854+0000 7f8e777fe700 1 -- 172.21.15.72:0/2497997394 learned_addr learned my addr 172.21.15.72:0/2497997394 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:55:51.746 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.854+0000 7f8e777fe700 1 -- 172.21.15.72:0/2497997394 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e6c009550 con 0x7f8e78104430 2024-03-26T01:55:51.746 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.854+0000 7f8e777fe700 1 --2- 172.21.15.72:0/2497997394 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e78104430 0x7f8e78199e50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c009130 tx=0x7f8e6c011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:55:51.746 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.855+0000 7f8e74ff9700 1 -- 172.21.15.72:0/2497997394 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e6c003aa0 con 0x7f8e78104430 2024-03-26T01:55:51.746 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.855+0000 7f8e74ff9700 1 -- 172.21.15.72:0/2497997394 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f8e6c01a8c0 con 0x7f8e78104430 2024-03-26T01:55:51.746 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.855+0000 7f8e7e18c700 1 -- 172.21.15.72:0/2497997394 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e7819a590 con 0x7f8e78104430 2024-03-26T01:55:51.747 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.855+0000 7f8e74ff9700 1 -- 172.21.15.72:0/2497997394 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8e6c019790 con 0x7f8e78104430 2024-03-26T01:55:51.747 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.855+0000 7f8e7e18c700 1 -- 172.21.15.72:0/2497997394 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e7819aa30 con 0x7f8e78104430 2024-03-26T01:55:51.747 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.855+0000 7f8e74ff9700 1 -- 172.21.15.72:0/2497997394 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 4) v1 ==== 45882+0+0 (secure 0 0 0) 0x7f8e6c019a50 con 0x7f8e78104430 2024-03-26T01:55:51.747 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.856+0000 7f8e74ff9700 1 --2- 172.21.15.72:0/2497997394 >> v2:172.21.15.72:6800/347688983 conn(0x7f8e60038e50 0x7f8e6003b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:55:51.747 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.856+0000 7f8e74ff9700 1 -- 172.21.15.72:0/2497997394 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f8e6c04d390 con 0x7f8e78104430 2024-03-26T01:55:51.747 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.856+0000 7f8e76ffd700 1 --2- 172.21.15.72:0/2497997394 >> v2:172.21.15.72:6800/347688983 conn(0x7f8e60038e50 0x7f8e6003b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:55:51.747 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.856+0000 7f8e7e18c700 1 -- 172.21.15.72:0/2497997394 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e780636b0 con 0x7f8e78104430 2024-03-26T01:55:51.747 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.856+0000 7f8e76ffd700 1 --2- 172.21.15.72:0/2497997394 >> v2:172.21.15.72:6800/347688983 conn(0x7f8e60038e50 0x7f8e6003b310 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8e68006fd0 tx=0x7f8e68006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:55:51.747 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:50.861+0000 7f8e74ff9700 1 -- 172.21.15.72:0/2497997394 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f8e6c018c30 con 0x7f8e78104430 2024-03-26T01:55:51.747 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:51.130+0000 7f8e7e18c700 1 -- 172.21.15.72:0/2497997394 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f8e7810a600 con 0x7f8e78104430 2024-03-26T01:55:51.747 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:51.130+0000 7f8e74ff9700 1 -- 172.21.15.72:0/2497997394 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f8e6c04c090 con 0x7f8e78104430 2024-03-26T01:55:51.748 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:51.132+0000 7f8e7e18c700 1 -- 172.21.15.72:0/2497997394 >> v2:172.21.15.72:6800/347688983 conn(0x7f8e60038e50 msgr2=0x7f8e6003b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:55:51.748 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:51.132+0000 7f8e7e18c700 1 --2- 172.21.15.72:0/2497997394 >> v2:172.21.15.72:6800/347688983 conn(0x7f8e60038e50 0x7f8e6003b310 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8e68006fd0 tx=0x7f8e68006e40 comp rx=0 tx=0).stop 2024-03-26T01:55:51.748 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:51.132+0000 7f8e7e18c700 1 -- 172.21.15.72:0/2497997394 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e78104430 msgr2=0x7f8e78199e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:55:51.748 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:51.132+0000 7f8e7e18c700 1 --2- 172.21.15.72:0/2497997394 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e78104430 0x7f8e78199e50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c009130 tx=0x7f8e6c011a50 comp rx=0 tx=0).stop 2024-03-26T01:55:51.748 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:51.133+0000 7f8e7e18c700 1 -- 172.21.15.72:0/2497997394 shutdown_connections 2024-03-26T01:55:51.748 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:51.133+0000 7f8e7e18c700 1 --2- 172.21.15.72:0/2497997394 >> v2:172.21.15.72:6800/347688983 conn(0x7f8e60038e50 0x7f8e6003b310 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:51.748 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:51.133+0000 7f8e7e18c700 1 --2- 172.21.15.72:0/2497997394 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8e78104430 0x7f8e78199e50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:51.748 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:51.133+0000 7f8e7e18c700 1 -- 172.21.15.72:0/2497997394 >> 172.21.15.72:0/2497997394 conn(0x7f8e780ffd10 msgr2=0x7f8e78100850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:55:51.748 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:51.133+0000 7f8e7e18c700 1 -- 172.21.15.72:0/2497997394 shutdown_connections 2024-03-26T01:55:51.748 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:51.133+0000 7f8e7e18c700 1 -- 172.21.15.72:0/2497997394 wait complete. 2024-03-26T01:55:51.748 INFO:teuthology.orchestra.run.smithi072.stdout:mgr is available 2024-03-26T01:55:52.689 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:52 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1783146505' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T01:55:52.961 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-03-26T01:55:52.961 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T01:55:52.961 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout fsid = c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T01:55:52.961 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T01:55:52.961 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T01:55:52.961 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T01:55:52.961 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T01:55:52.961 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-03-26T01:55:52.962 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T01:55:52.962 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T01:55:52.962 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 2024-03-26T01:55:52.962 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T01:55:52.962 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T01:55:52.962 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T01:55:52.962 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.181+0000 7fb853ae9700 1 Processor -- start 2024-03-26T01:55:52.962 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.181+0000 7fb853ae9700 1 -- start start 2024-03-26T01:55:52.962 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.181+0000 7fb853ae9700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb84c1060f0 0x7fb84c106510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:55:52.962 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.181+0000 7fb853ae9700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c106ae0 con 0x7fb84c1060f0 2024-03-26T01:55:52.962 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.181+0000 7fb851885700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb84c1060f0 0x7fb84c106510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:55:52.962 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.181+0000 7fb851885700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb84c1060f0 0x7fb84c106510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58010/0 (socket says 172.21.15.72:58010) 2024-03-26T01:55:52.962 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.181+0000 7fb851885700 1 -- 172.21.15.72:0/4160153578 learned_addr learned my addr 172.21.15.72:0/4160153578 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:55:52.962 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.182+0000 7fb851885700 1 -- 172.21.15.72:0/4160153578 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb84c1072f0 con 0x7fb84c1060f0 2024-03-26T01:55:52.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.182+0000 7fb851885700 1 --2- 172.21.15.72:0/4160153578 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb84c1060f0 0x7fb84c106510 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb8480098a0 tx=0x7fb848009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c0421f0d691d9b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:55:52.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.182+0000 7fb850883700 1 -- 172.21.15.72:0/4160153578 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb848004160 con 0x7fb84c1060f0 2024-03-26T01:55:52.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.182+0000 7fb850883700 1 -- 172.21.15.72:0/4160153578 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7fb8480042c0 con 0x7fb84c1060f0 2024-03-26T01:55:52.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.182+0000 7fb850883700 1 -- 172.21.15.72:0/4160153578 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8480119b0 con 0x7fb84c1060f0 2024-03-26T01:55:52.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.182+0000 7fb853ae9700 1 -- 172.21.15.72:0/4160153578 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb84c1060f0 msgr2=0x7fb84c106510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:55:52.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.182+0000 7fb853ae9700 1 --2- 172.21.15.72:0/4160153578 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb84c1060f0 0x7fb84c106510 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb8480098a0 tx=0x7fb848009bb0 comp rx=0 tx=0).stop 2024-03-26T01:55:52.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.182+0000 7fb853ae9700 1 -- 172.21.15.72:0/4160153578 shutdown_connections 2024-03-26T01:55:52.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.182+0000 7fb853ae9700 1 --2- 172.21.15.72:0/4160153578 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb84c1060f0 0x7fb84c106510 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:52.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.182+0000 7fb853ae9700 1 -- 172.21.15.72:0/4160153578 >> 172.21.15.72:0/4160153578 conn(0x7fb84c1016b0 msgr2=0x7fb84c103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:55:52.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.183+0000 7fb853ae9700 1 -- 172.21.15.72:0/4160153578 shutdown_connections 2024-03-26T01:55:52.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.183+0000 7fb853ae9700 1 -- 172.21.15.72:0/4160153578 wait complete. 2024-03-26T01:55:52.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.183+0000 7fb853ae9700 1 Processor -- start 2024-03-26T01:55:52.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.183+0000 7fb853ae9700 1 -- start start 2024-03-26T01:55:52.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.183+0000 7fb853ae9700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb84c1060f0 0x7fb84c199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:55:52.963 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.183+0000 7fb853ae9700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb84c19a3b0 con 0x7fb84c1060f0 2024-03-26T01:55:52.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.183+0000 7fb851885700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb84c1060f0 0x7fb84c199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:55:52.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.183+0000 7fb851885700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb84c1060f0 0x7fb84c199e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58012/0 (socket says 172.21.15.72:58012) 2024-03-26T01:55:52.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.183+0000 7fb851885700 1 -- 172.21.15.72:0/1783146505 learned_addr learned my addr 172.21.15.72:0/1783146505 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:55:52.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.184+0000 7fb851885700 1 -- 172.21.15.72:0/1783146505 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb848009550 con 0x7fb84c1060f0 2024-03-26T01:55:52.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.184+0000 7fb851885700 1 --2- 172.21.15.72:0/1783146505 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb84c1060f0 0x7fb84c199e70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb848006af0 tx=0x7fb848011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:55:52.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.184+0000 7fb842ffd700 1 -- 172.21.15.72:0/1783146505 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8480039e0 con 0x7fb84c1060f0 2024-03-26T01:55:52.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.184+0000 7fb842ffd700 1 -- 172.21.15.72:0/1783146505 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7fb848003b40 con 0x7fb84c1060f0 2024-03-26T01:55:52.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.184+0000 7fb853ae9700 1 -- 172.21.15.72:0/1783146505 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb84c19a5b0 con 0x7fb84c1060f0 2024-03-26T01:55:52.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.184+0000 7fb842ffd700 1 -- 172.21.15.72:0/1783146505 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb848019720 con 0x7fb84c1060f0 2024-03-26T01:55:52.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.184+0000 7fb853ae9700 1 -- 172.21.15.72:0/1783146505 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb84c19aa50 con 0x7fb84c1060f0 2024-03-26T01:55:52.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.185+0000 7fb842ffd700 1 -- 172.21.15.72:0/1783146505 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 4) v1 ==== 45882+0+0 (secure 0 0 0) 0x7fb848019950 con 0x7fb84c1060f0 2024-03-26T01:55:52.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.185+0000 7fb853ae9700 1 -- 172.21.15.72:0/1783146505 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb84c0636b0 con 0x7fb84c1060f0 2024-03-26T01:55:52.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.185+0000 7fb842ffd700 1 --2- 172.21.15.72:0/1783146505 >> v2:172.21.15.72:6800/347688983 conn(0x7fb838038ea0 0x7fb83803b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:55:52.964 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.185+0000 7fb842ffd700 1 -- 172.21.15.72:0/1783146505 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb84804cfe0 con 0x7fb84c1060f0 2024-03-26T01:55:52.965 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.185+0000 7fb851084700 1 --2- 172.21.15.72:0/1783146505 >> v2:172.21.15.72:6800/347688983 conn(0x7fb838038ea0 0x7fb83803b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:55:52.965 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.185+0000 7fb851084700 1 --2- 172.21.15.72:0/1783146505 >> v2:172.21.15.72:6800/347688983 conn(0x7fb838038ea0 0x7fb83803b360 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb83c006fd0 tx=0x7fb83c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:55:52.965 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.190+0000 7fb842ffd700 1 -- 172.21.15.72:0/1783146505 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb848019c00 con 0x7fb84c1060f0 2024-03-26T01:55:52.965 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.379+0000 7fb853ae9700 1 -- 172.21.15.72:0/1783146505 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fb84c10a520 con 0x7fb84c1060f0 2024-03-26T01:55:52.965 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.380+0000 7fb842ffd700 1 -- 172.21.15.72:0/1783146505 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+282 (secure 0 0 0) 0x7fb8480108d0 con 0x7fb84c1060f0 2024-03-26T01:55:52.965 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.382+0000 7fb853ae9700 1 -- 172.21.15.72:0/1783146505 >> v2:172.21.15.72:6800/347688983 conn(0x7fb838038ea0 msgr2=0x7fb83803b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:55:52.965 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.382+0000 7fb853ae9700 1 --2- 172.21.15.72:0/1783146505 >> v2:172.21.15.72:6800/347688983 conn(0x7fb838038ea0 0x7fb83803b360 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb83c006fd0 tx=0x7fb83c006e40 comp rx=0 tx=0).stop 2024-03-26T01:55:52.965 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.382+0000 7fb853ae9700 1 -- 172.21.15.72:0/1783146505 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb84c1060f0 msgr2=0x7fb84c199e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:55:52.965 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.382+0000 7fb853ae9700 1 --2- 172.21.15.72:0/1783146505 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb84c1060f0 0x7fb84c199e70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb848006af0 tx=0x7fb848011a50 comp rx=0 tx=0).stop 2024-03-26T01:55:52.965 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.382+0000 7fb853ae9700 1 -- 172.21.15.72:0/1783146505 shutdown_connections 2024-03-26T01:55:52.965 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.382+0000 7fb853ae9700 1 --2- 172.21.15.72:0/1783146505 >> v2:172.21.15.72:6800/347688983 conn(0x7fb838038ea0 0x7fb83803b360 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:52.965 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.382+0000 7fb853ae9700 1 --2- 172.21.15.72:0/1783146505 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb84c1060f0 0x7fb84c199e70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:52.965 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.382+0000 7fb853ae9700 1 -- 172.21.15.72:0/1783146505 >> 172.21.15.72:0/1783146505 conn(0x7fb84c1016b0 msgr2=0x7fb84c102220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:55:52.965 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.382+0000 7fb853ae9700 1 -- 172.21.15.72:0/1783146505 shutdown_connections 2024-03-26T01:55:52.965 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:52.382+0000 7fb853ae9700 1 -- 172.21.15.72:0/1783146505 wait complete. 2024-03-26T01:55:52.966 INFO:teuthology.orchestra.run.smithi072.stdout:Enabling cephadm module... 2024-03-26T01:55:53.968 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:53 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3644498371' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-26T01:55:53.968 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:53 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ignoring --setuser ceph since I am not root 2024-03-26T01:55:53.969 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:53 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ignoring --setgroup ceph since I am not root 2024-03-26T01:55:54.306 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:54.265+0000 7f666a4fc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T01:55:54.315 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.405+0000 7f7ed8a82700 1 Processor -- start 2024-03-26T01:55:54.315 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.405+0000 7f7ed8a82700 1 -- start start 2024-03-26T01:55:54.315 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.405+0000 7f7ed8a82700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ed407bfa0 0x7f7ed407a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:55:54.315 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.405+0000 7f7ed8a82700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed407a9d0 con 0x7f7ed407bfa0 2024-03-26T01:55:54.315 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.405+0000 7f7ed259c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ed407bfa0 0x7f7ed407a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:55:54.315 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.405+0000 7f7ed259c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ed407bfa0 0x7f7ed407a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58020/0 (socket says 172.21.15.72:58020) 2024-03-26T01:55:54.315 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.405+0000 7f7ed259c700 1 -- 172.21.15.72:0/2325503113 learned_addr learned my addr 172.21.15.72:0/2325503113 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:55:54.315 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.405+0000 7f7ed259c700 1 -- 172.21.15.72:0/2325503113 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ed407ab10 con 0x7f7ed407bfa0 2024-03-26T01:55:54.315 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.406+0000 7f7ed259c700 1 --2- 172.21.15.72:0/2325503113 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ed407bfa0 0x7f7ed407a400 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7ec40098a0 tx=0x7f7ec4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0725aed85989355 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:55:54.315 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.406+0000 7f7ed159a700 1 -- 172.21.15.72:0/2325503113 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ec4004160 con 0x7f7ed407bfa0 2024-03-26T01:55:54.315 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.406+0000 7f7ed159a700 1 -- 172.21.15.72:0/2325503113 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f7ec40042c0 con 0x7f7ed407bfa0 2024-03-26T01:55:54.315 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.406+0000 7f7ed159a700 1 -- 172.21.15.72:0/2325503113 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ec40119b0 con 0x7f7ed407bfa0 2024-03-26T01:55:54.316 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.406+0000 7f7ed8a82700 1 -- 172.21.15.72:0/2325503113 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ed407bfa0 msgr2=0x7f7ed407a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:55:54.316 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.406+0000 7f7ed8a82700 1 --2- 172.21.15.72:0/2325503113 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ed407bfa0 0x7f7ed407a400 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7ec40098a0 tx=0x7f7ec4009bb0 comp rx=0 tx=0).stop 2024-03-26T01:55:54.316 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.406+0000 7f7ed8a82700 1 -- 172.21.15.72:0/2325503113 shutdown_connections 2024-03-26T01:55:54.316 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.406+0000 7f7ed8a82700 1 --2- 172.21.15.72:0/2325503113 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ed407bfa0 0x7f7ed407a400 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:54.316 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.406+0000 7f7ed8a82700 1 -- 172.21.15.72:0/2325503113 >> 172.21.15.72:0/2325503113 conn(0x7f7ed41016b0 msgr2=0x7f7ed4103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:55:54.316 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.406+0000 7f7ed8a82700 1 -- 172.21.15.72:0/2325503113 shutdown_connections 2024-03-26T01:55:54.316 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.406+0000 7f7ed8a82700 1 -- 172.21.15.72:0/2325503113 wait complete. 2024-03-26T01:55:54.316 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.407+0000 7f7ed8a82700 1 Processor -- start 2024-03-26T01:55:54.317 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.407+0000 7f7ed8a82700 1 -- start start 2024-03-26T01:55:54.317 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.407+0000 7f7ed8a82700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ed407bfa0 0x7f7ed419c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:55:54.317 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.407+0000 7f7ed8a82700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ed419c5b0 con 0x7f7ed407bfa0 2024-03-26T01:55:54.317 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.407+0000 7f7ed259c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ed407bfa0 0x7f7ed419c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:55:54.317 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.407+0000 7f7ed259c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ed407bfa0 0x7f7ed419c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58028/0 (socket says 172.21.15.72:58028) 2024-03-26T01:55:54.318 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.407+0000 7f7ed259c700 1 -- 172.21.15.72:0/3644498371 learned_addr learned my addr 172.21.15.72:0/3644498371 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:55:54.318 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.408+0000 7f7ed259c700 1 -- 172.21.15.72:0/3644498371 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec4009550 con 0x7f7ed407bfa0 2024-03-26T01:55:54.318 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.408+0000 7f7ed259c700 1 --2- 172.21.15.72:0/3644498371 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ed407bfa0 0x7f7ed419c070 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7ec4006af0 tx=0x7f7ec4011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:55:54.318 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.408+0000 7f7ec37fe700 1 -- 172.21.15.72:0/3644498371 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ec40039e0 con 0x7f7ed407bfa0 2024-03-26T01:55:54.318 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.408+0000 7f7ec37fe700 1 -- 172.21.15.72:0/3644498371 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f7ec4003b40 con 0x7f7ed407bfa0 2024-03-26T01:55:54.318 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.408+0000 7f7ed8a82700 1 -- 172.21.15.72:0/3644498371 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ed419c7b0 con 0x7f7ed407bfa0 2024-03-26T01:55:54.318 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.408+0000 7f7ec37fe700 1 -- 172.21.15.72:0/3644498371 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ec4019720 con 0x7f7ed407bfa0 2024-03-26T01:55:54.318 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.408+0000 7f7ed8a82700 1 -- 172.21.15.72:0/3644498371 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ed419cc50 con 0x7f7ed407bfa0 2024-03-26T01:55:54.318 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.409+0000 7f7ec37fe700 1 -- 172.21.15.72:0/3644498371 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 4) v1 ==== 45882+0+0 (secure 0 0 0) 0x7f7ec40199a0 con 0x7f7ed407bfa0 2024-03-26T01:55:54.318 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.409+0000 7f7ed8a82700 1 -- 172.21.15.72:0/3644498371 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ed410c4c0 con 0x7f7ed407bfa0 2024-03-26T01:55:54.318 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.409+0000 7f7ec37fe700 1 --2- 172.21.15.72:0/3644498371 >> v2:172.21.15.72:6800/347688983 conn(0x7f7ebc038e50 0x7f7ebc03b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:55:54.319 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.409+0000 7f7ec37fe700 1 -- 172.21.15.72:0/3644498371 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f7ec404cea0 con 0x7f7ed407bfa0 2024-03-26T01:55:54.319 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.409+0000 7f7ed1d9b700 1 --2- 172.21.15.72:0/3644498371 >> v2:172.21.15.72:6800/347688983 conn(0x7f7ebc038e50 0x7f7ebc03b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:55:54.319 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.410+0000 7f7ed1d9b700 1 --2- 172.21.15.72:0/3644498371 >> v2:172.21.15.72:6800/347688983 conn(0x7f7ebc038e50 0x7f7ebc03b310 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8006fd0 tx=0x7f7ec8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:55:54.319 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.414+0000 7f7ec37fe700 1 -- 172.21.15.72:0/3644498371 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f7ec4010a70 con 0x7f7ed407bfa0 2024-03-26T01:55:54.319 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.660+0000 7f7ed8a82700 1 -- 172.21.15.72:0/3644498371 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f7ed419f590 con 0x7f7ed407bfa0 2024-03-26T01:55:54.319 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.712+0000 7f7ec37fe700 1 -- 172.21.15.72:0/3644498371 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f7ec4010380 con 0x7f7ed407bfa0 2024-03-26T01:55:54.319 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.713+0000 7f7ec37fe700 1 -- 172.21.15.72:0/3644498371 <== mon.0 v2:172.21.15.72:3300/0 8 ==== mgrmap(e 5) v1 ==== 45893+0+0 (secure 0 0 0) 0x7f7ec402c960 con 0x7f7ed407bfa0 2024-03-26T01:55:54.319 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.715+0000 7f7ed8a82700 1 -- 172.21.15.72:0/3644498371 >> v2:172.21.15.72:6800/347688983 conn(0x7f7ebc038e50 msgr2=0x7f7ebc03b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:55:54.319 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.715+0000 7f7ed8a82700 1 --2- 172.21.15.72:0/3644498371 >> v2:172.21.15.72:6800/347688983 conn(0x7f7ebc038e50 0x7f7ebc03b310 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8006fd0 tx=0x7f7ec8006e40 comp rx=0 tx=0).stop 2024-03-26T01:55:54.319 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.715+0000 7f7ed8a82700 1 -- 172.21.15.72:0/3644498371 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ed407bfa0 msgr2=0x7f7ed419c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:55:54.319 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.715+0000 7f7ed8a82700 1 --2- 172.21.15.72:0/3644498371 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ed407bfa0 0x7f7ed419c070 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7ec4006af0 tx=0x7f7ec4011a50 comp rx=0 tx=0).stop 2024-03-26T01:55:54.319 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.715+0000 7f7ed8a82700 1 -- 172.21.15.72:0/3644498371 shutdown_connections 2024-03-26T01:55:54.319 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.715+0000 7f7ed8a82700 1 --2- 172.21.15.72:0/3644498371 >> v2:172.21.15.72:6800/347688983 conn(0x7f7ebc038e50 0x7f7ebc03b310 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:54.320 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.715+0000 7f7ed8a82700 1 --2- 172.21.15.72:0/3644498371 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7ed407bfa0 0x7f7ed419c070 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:54.320 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.715+0000 7f7ed8a82700 1 -- 172.21.15.72:0/3644498371 >> 172.21.15.72:0/3644498371 conn(0x7f7ed41016b0 msgr2=0x7f7ed4102210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:55:54.320 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.715+0000 7f7ed8a82700 1 -- 172.21.15.72:0/3644498371 shutdown_connections 2024-03-26T01:55:54.320 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:53.715+0000 7f7ed8a82700 1 -- 172.21.15.72:0/3644498371 wait complete. 2024-03-26T01:55:54.567 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:54.362+0000 7f666a4fc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T01:55:54.968 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:54 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3644498371' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-26T01:55:54.968 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:54 smithi072 ceph-mon[30177]: mgrmap e5: y(active, since 6s) 2024-03-26T01:55:55.252 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:55 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:55.007+0000 7f666a4fc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T01:55:55.252 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:55 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:55.109+0000 7f666a4fc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T01:55:55.252 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:55 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:55.251+0000 7f666a4fc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T01:55:55.639 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:55 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:55.348+0000 7f666a4fc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T01:55:55.639 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:55 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:55.431+0000 7f666a4fc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T01:55:55.646 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout { 2024-03-26T01:55:55.647 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-26T01:55:55.647 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T01:55:55.647 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-26T01:55:55.647 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T01:55:55.647 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout } 2024-03-26T01:55:55.647 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.778+0000 7ff884e5e700 1 Processor -- start 2024-03-26T01:55:55.647 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.778+0000 7ff884e5e700 1 -- start start 2024-03-26T01:55:55.647 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.779+0000 7ff884e5e700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff880106b80 0x7ff880108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:55:55.647 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.779+0000 7ff884e5e700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff880079f70 con 0x7ff880106b80 2024-03-26T01:55:55.647 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.779+0000 7ff87e59c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff880106b80 0x7ff880108f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:55:55.647 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.779+0000 7ff87e59c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff880106b80 0x7ff880108f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58050/0 (socket says 172.21.15.72:58050) 2024-03-26T01:55:55.647 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.779+0000 7ff87e59c700 1 -- 172.21.15.72:0/120973821 learned_addr learned my addr 172.21.15.72:0/120973821 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:55:55.647 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.779+0000 7ff87e59c700 1 -- 172.21.15.72:0/120973821 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8801094b0 con 0x7ff880106b80 2024-03-26T01:55:55.647 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.779+0000 7ff87e59c700 1 --2- 172.21.15.72:0/120973821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff880106b80 0x7ff880108f70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff8700098a0 tx=0x7ff870009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e1cf42be5526f71 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:55:55.648 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.779+0000 7ff87d59a700 1 -- 172.21.15.72:0/120973821 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff870004160 con 0x7ff880106b80 2024-03-26T01:55:55.648 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.779+0000 7ff87d59a700 1 -- 172.21.15.72:0/120973821 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7ff8700042c0 con 0x7ff880106b80 2024-03-26T01:55:55.648 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.779+0000 7ff87d59a700 1 -- 172.21.15.72:0/120973821 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff8700119b0 con 0x7ff880106b80 2024-03-26T01:55:55.648 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.780+0000 7ff884e5e700 1 -- 172.21.15.72:0/120973821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff880106b80 msgr2=0x7ff880108f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:55:55.648 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.780+0000 7ff884e5e700 1 --2- 172.21.15.72:0/120973821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff880106b80 0x7ff880108f70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff8700098a0 tx=0x7ff870009bb0 comp rx=0 tx=0).stop 2024-03-26T01:55:55.648 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.780+0000 7ff884e5e700 1 -- 172.21.15.72:0/120973821 shutdown_connections 2024-03-26T01:55:55.648 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.780+0000 7ff884e5e700 1 --2- 172.21.15.72:0/120973821 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff880106b80 0x7ff880108f70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:55.648 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.780+0000 7ff884e5e700 1 -- 172.21.15.72:0/120973821 >> 172.21.15.72:0/120973821 conn(0x7ff880100550 msgr2=0x7ff8801029b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:55:55.648 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.780+0000 7ff884e5e700 1 -- 172.21.15.72:0/120973821 shutdown_connections 2024-03-26T01:55:55.648 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.780+0000 7ff884e5e700 1 -- 172.21.15.72:0/120973821 wait complete. 2024-03-26T01:55:55.648 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.781+0000 7ff884e5e700 1 Processor -- start 2024-03-26T01:55:55.648 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.781+0000 7ff884e5e700 1 -- start start 2024-03-26T01:55:55.648 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.781+0000 7ff884e5e700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff880106b80 0x7ff880199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:55:55.648 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.781+0000 7ff884e5e700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff88019a340 con 0x7ff880106b80 2024-03-26T01:55:55.649 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.781+0000 7ff87e59c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff880106b80 0x7ff880199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:55:55.649 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.781+0000 7ff87e59c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff880106b80 0x7ff880199e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58062/0 (socket says 172.21.15.72:58062) 2024-03-26T01:55:55.649 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.781+0000 7ff87e59c700 1 -- 172.21.15.72:0/184021934 learned_addr learned my addr 172.21.15.72:0/184021934 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:55:55.649 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.782+0000 7ff87e59c700 1 -- 172.21.15.72:0/184021934 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff870009550 con 0x7ff880106b80 2024-03-26T01:55:55.649 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.782+0000 7ff87e59c700 1 --2- 172.21.15.72:0/184021934 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff880106b80 0x7ff880199e00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff870009130 tx=0x7ff870011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:55:55.649 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.782+0000 7ff86f7fe700 1 -- 172.21.15.72:0/184021934 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff870003aa0 con 0x7ff880106b80 2024-03-26T01:55:55.649 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.782+0000 7ff884e5e700 1 -- 172.21.15.72:0/184021934 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff88019a540 con 0x7ff880106b80 2024-03-26T01:55:55.649 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.782+0000 7ff86f7fe700 1 -- 172.21.15.72:0/184021934 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7ff87001a8c0 con 0x7ff880106b80 2024-03-26T01:55:55.649 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.782+0000 7ff86f7fe700 1 -- 172.21.15.72:0/184021934 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff870019790 con 0x7ff880106b80 2024-03-26T01:55:55.649 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.782+0000 7ff884e5e700 1 -- 172.21.15.72:0/184021934 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff88019a9e0 con 0x7ff880106b80 2024-03-26T01:55:55.649 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.783+0000 7ff86f7fe700 1 -- 172.21.15.72:0/184021934 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 45893+0+0 (secure 0 0 0) 0x7ff870019a00 con 0x7ff880106b80 2024-03-26T01:55:55.649 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.783+0000 7ff884e5e700 1 -- 172.21.15.72:0/184021934 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff88010a250 con 0x7ff880106b80 2024-03-26T01:55:55.649 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.783+0000 7ff86f7fe700 1 --2- 172.21.15.72:0/184021934 >> v2:172.21.15.72:6800/347688983 conn(0x7ff868038ef0 0x7ff86803b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:55:55.649 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.783+0000 7ff86f7fe700 1 -- 172.21.15.72:0/184021934 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff87004e140 con 0x7ff880106b80 2024-03-26T01:55:55.649 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.783+0000 7ff87dd9b700 1 -- 172.21.15.72:0/184021934 >> v2:172.21.15.72:6800/347688983 conn(0x7ff868038ef0 msgr2=0x7ff86803b3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/347688983 2024-03-26T01:55:55.650 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.783+0000 7ff87dd9b700 1 --2- 172.21.15.72:0/184021934 >> v2:172.21.15.72:6800/347688983 conn(0x7ff868038ef0 0x7ff86803b3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T01:55:55.650 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.788+0000 7ff86f7fe700 1 -- 172.21.15.72:0/184021934 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff870018c30 con 0x7ff880106b80 2024-03-26T01:55:55.650 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.984+0000 7ff87dd9b700 1 -- 172.21.15.72:0/184021934 >> v2:172.21.15.72:6800/347688983 conn(0x7ff868038ef0 msgr2=0x7ff86803b3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/347688983 2024-03-26T01:55:55.650 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:54.984+0000 7ff87dd9b700 1 --2- 172.21.15.72:0/184021934 >> v2:172.21.15.72:6800/347688983 conn(0x7ff868038ef0 0x7ff86803b3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T01:55:55.650 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:55.041+0000 7ff884e5e700 1 -- 172.21.15.72:0/184021934 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7ff88019d3c0 con 0x7ff880106b80 2024-03-26T01:55:55.650 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:55.042+0000 7ff86f7fe700 1 -- 172.21.15.72:0/184021934 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7ff870018e60 con 0x7ff880106b80 2024-03-26T01:55:55.650 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:55.045+0000 7ff884e5e700 1 -- 172.21.15.72:0/184021934 >> v2:172.21.15.72:6800/347688983 conn(0x7ff868038ef0 msgr2=0x7ff86803b3b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T01:55:55.650 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:55.045+0000 7ff884e5e700 1 --2- 172.21.15.72:0/184021934 >> v2:172.21.15.72:6800/347688983 conn(0x7ff868038ef0 0x7ff86803b3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:55.650 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:55.045+0000 7ff884e5e700 1 -- 172.21.15.72:0/184021934 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff880106b80 msgr2=0x7ff880199e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:55:55.650 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:55.045+0000 7ff884e5e700 1 --2- 172.21.15.72:0/184021934 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff880106b80 0x7ff880199e00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff870009130 tx=0x7ff870011a50 comp rx=0 tx=0).stop 2024-03-26T01:55:55.650 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:55.045+0000 7ff884e5e700 1 -- 172.21.15.72:0/184021934 shutdown_connections 2024-03-26T01:55:55.650 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:55.045+0000 7ff884e5e700 1 --2- 172.21.15.72:0/184021934 >> v2:172.21.15.72:6800/347688983 conn(0x7ff868038ef0 0x7ff86803b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:55.650 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:55.045+0000 7ff884e5e700 1 --2- 172.21.15.72:0/184021934 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff880106b80 0x7ff880199e00 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:55:55.651 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:55.045+0000 7ff884e5e700 1 -- 172.21.15.72:0/184021934 >> 172.21.15.72:0/184021934 conn(0x7ff880100550 msgr2=0x7ff8801010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:55:55.651 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:55.045+0000 7ff884e5e700 1 -- 172.21.15.72:0/184021934 shutdown_connections 2024-03-26T01:55:55.651 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:55.045+0000 7ff884e5e700 1 -- 172.21.15.72:0/184021934 wait complete. 2024-03-26T01:55:55.651 INFO:teuthology.orchestra.run.smithi072.stdout:Waiting for the mgr to restart... 2024-03-26T01:55:55.651 INFO:teuthology.orchestra.run.smithi072.stdout:Waiting for mgr epoch 5... 2024-03-26T01:55:55.889 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:55:55 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/184021934' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T01:55:55.889 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:55 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:55.702+0000 7f666a4fc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T01:55:55.889 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:55 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:55.812+0000 7f666a4fc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T01:55:56.140 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:55 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:55.902+0000 7f666a4fc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T01:55:56.141 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:56 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:55.998+0000 7f666a4fc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T01:55:56.468 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:56 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:56.178+0000 7f666a4fc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T01:55:57.468 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:57 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:57.127+0000 7f666a4fc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T01:55:57.468 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:57 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:57.254+0000 7f666a4fc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T01:55:57.867 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:57 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:57.583+0000 7f666a4fc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T01:55:57.867 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:57 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:57.672+0000 7f666a4fc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T01:55:57.867 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:57 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:57.866+0000 7f666a4fc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T01:55:58.218 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:58.049+0000 7f666a4fc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T01:55:58.584 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:58.329+0000 7f666a4fc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T01:55:58.585 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:58.584+0000 7f666a4fc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T01:55:58.968 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:58.816+0000 7f666a4fc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T01:55:59.968 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:55:59 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:55:59.598+0000 7f666a4fc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T01:56:00.468 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:00 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:00.159+0000 7f666a4fc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T01:56:00.468 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:00 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:00.256+0000 7f666a4fc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T01:56:00.968 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:00 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:00.469+0000 7f666a4fc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T01:56:00.968 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:00 smithi072 ceph-mon[30177]: Active manager daemon y restarted 2024-03-26T01:56:00.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:00 smithi072 ceph-mon[30177]: Activating manager daemon y 2024-03-26T01:56:00.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:00 smithi072 ceph-mon[30177]: osdmap e2: 0 total, 0 up, 0 in 2024-03-26T01:56:00.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:00 smithi072 ceph-mon[30177]: mgrmap e6: y(active, starting, since 0.00338856s) 2024-03-26T01:56:00.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:00 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:56:00.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:00 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T01:56:00.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:00 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T01:56:00.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:00 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T01:56:00.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:00 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T01:56:00.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:00 smithi072 ceph-mon[30177]: Manager daemon y is now available 2024-03-26T01:56:00.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:00 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' 2024-03-26T01:56:00.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:00 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' 2024-03-26T01:56:00.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:00 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:00.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:00 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:01.783 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:01 smithi072 ceph-mon[30177]: Found migration_current of "None". Setting to last migration. 2024-03-26T01:56:01.783 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:01 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:01.783 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:01 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:56:01.783 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:01 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T01:56:01.783 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:01 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' 2024-03-26T01:56:01.783 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:01 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' 2024-03-26T01:56:01.783 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:01 smithi072 ceph-mon[30177]: mgrmap e7: y(active, since 1.00553s) 2024-03-26T01:56:01.783 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:01 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:02.079 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout { 2024-03-26T01:56:02.080 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-26T01:56:02.080 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T01:56:02.080 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout } 2024-03-26T01:56:02.080 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.139+0000 7eff521a1700 1 Processor -- start 2024-03-26T01:56:02.080 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.139+0000 7eff521a1700 1 -- start start 2024-03-26T01:56:02.080 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.139+0000 7eff521a1700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff4c104e00 0x7eff4c105220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:02.080 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.140+0000 7eff521a1700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4c105760 con 0x7eff4c104e00 2024-03-26T01:56:02.080 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.140+0000 7eff5119f700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff4c104e00 0x7eff4c105220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:02.080 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.140+0000 7eff5119f700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff4c104e00 0x7eff4c105220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58074/0 (socket says 172.21.15.72:58074) 2024-03-26T01:56:02.080 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.140+0000 7eff5119f700 1 -- 172.21.15.72:0/2504201758 learned_addr learned my addr 172.21.15.72:0/2504201758 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:02.080 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.140+0000 7eff5119f700 1 -- 172.21.15.72:0/2504201758 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff4c1058a0 con 0x7eff4c104e00 2024-03-26T01:56:02.080 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.140+0000 7eff5119f700 1 --2- 172.21.15.72:0/2504201758 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff4c104e00 0x7eff4c105220 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0098a0 tx=0x7eff3c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a0e50b9fb733234 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:02.080 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.140+0000 7eff43fff700 1 -- 172.21.15.72:0/2504201758 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff3c004160 con 0x7eff4c104e00 2024-03-26T01:56:02.081 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.140+0000 7eff43fff700 1 -- 172.21.15.72:0/2504201758 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7eff3c0042c0 con 0x7eff4c104e00 2024-03-26T01:56:02.081 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.140+0000 7eff43fff700 1 -- 172.21.15.72:0/2504201758 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff3c0119b0 con 0x7eff4c104e00 2024-03-26T01:56:02.081 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.141+0000 7eff521a1700 1 -- 172.21.15.72:0/2504201758 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff4c104e00 msgr2=0x7eff4c105220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:02.081 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.141+0000 7eff521a1700 1 --2- 172.21.15.72:0/2504201758 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff4c104e00 0x7eff4c105220 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0098a0 tx=0x7eff3c009bb0 comp rx=0 tx=0).stop 2024-03-26T01:56:02.081 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.141+0000 7eff521a1700 1 -- 172.21.15.72:0/2504201758 shutdown_connections 2024-03-26T01:56:02.081 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.141+0000 7eff521a1700 1 --2- 172.21.15.72:0/2504201758 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff4c104e00 0x7eff4c105220 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:02.081 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.141+0000 7eff521a1700 1 -- 172.21.15.72:0/2504201758 >> 172.21.15.72:0/2504201758 conn(0x7eff4c100380 msgr2=0x7eff4c1027e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:02.081 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.141+0000 7eff521a1700 1 -- 172.21.15.72:0/2504201758 shutdown_connections 2024-03-26T01:56:02.081 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.141+0000 7eff521a1700 1 -- 172.21.15.72:0/2504201758 wait complete. 2024-03-26T01:56:02.082 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.142+0000 7eff521a1700 1 Processor -- start 2024-03-26T01:56:02.082 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.142+0000 7eff521a1700 1 -- start start 2024-03-26T01:56:02.082 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.142+0000 7eff521a1700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff4c104e00 0x7eff4c199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:02.082 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.142+0000 7eff521a1700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4c19a1f0 con 0x7eff4c104e00 2024-03-26T01:56:02.082 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.142+0000 7eff5119f700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff4c104e00 0x7eff4c199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:02.082 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.142+0000 7eff5119f700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff4c104e00 0x7eff4c199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:58086/0 (socket says 172.21.15.72:58086) 2024-03-26T01:56:02.082 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.142+0000 7eff5119f700 1 -- 172.21.15.72:0/4044356141 learned_addr learned my addr 172.21.15.72:0/4044356141 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:02.082 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.143+0000 7eff5119f700 1 -- 172.21.15.72:0/4044356141 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff3c009550 con 0x7eff4c104e00 2024-03-26T01:56:02.082 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.143+0000 7eff5119f700 1 --2- 172.21.15.72:0/4044356141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff4c104e00 0x7eff4c199cb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0046c0 tx=0x7eff3c011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:02.082 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.143+0000 7eff427fc700 1 -- 172.21.15.72:0/4044356141 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff3c0039c0 con 0x7eff4c104e00 2024-03-26T01:56:02.082 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.143+0000 7eff427fc700 1 -- 172.21.15.72:0/4044356141 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7eff3c003b20 con 0x7eff4c104e00 2024-03-26T01:56:02.082 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.143+0000 7eff521a1700 1 -- 172.21.15.72:0/4044356141 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff4c19a3f0 con 0x7eff4c104e00 2024-03-26T01:56:02.082 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.143+0000 7eff427fc700 1 -- 172.21.15.72:0/4044356141 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff3c01a7b0 con 0x7eff4c104e00 2024-03-26T01:56:02.083 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.143+0000 7eff521a1700 1 -- 172.21.15.72:0/4044356141 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff4c19a890 con 0x7eff4c104e00 2024-03-26T01:56:02.083 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.144+0000 7eff427fc700 1 -- 172.21.15.72:0/4044356141 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 5) v1 ==== 45893+0+0 (secure 0 0 0) 0x7eff3c024430 con 0x7eff4c104e00 2024-03-26T01:56:02.083 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.144+0000 7eff427fc700 1 --2- 172.21.15.72:0/4044356141 >> v2:172.21.15.72:6800/347688983 conn(0x7eff38038ea0 0x7eff3803b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:02.083 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.144+0000 7eff427fc700 1 -- 172.21.15.72:0/4044356141 --> v2:172.21.15.72:6800/347688983 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7eff3803ba30 con 0x7eff38038ea0 2024-03-26T01:56:02.083 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.144+0000 7eff5099e700 1 -- 172.21.15.72:0/4044356141 >> v2:172.21.15.72:6800/347688983 conn(0x7eff38038ea0 msgr2=0x7eff3803b360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/347688983 2024-03-26T01:56:02.083 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.144+0000 7eff427fc700 1 -- 172.21.15.72:0/4044356141 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7eff3c04cff0 con 0x7eff4c104e00 2024-03-26T01:56:02.084 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.144+0000 7eff5099e700 1 --2- 172.21.15.72:0/4044356141 >> v2:172.21.15.72:6800/347688983 conn(0x7eff38038ea0 0x7eff3803b360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T01:56:02.084 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.345+0000 7eff5099e700 1 -- 172.21.15.72:0/4044356141 >> v2:172.21.15.72:6800/347688983 conn(0x7eff38038ea0 msgr2=0x7eff3803b360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/347688983 2024-03-26T01:56:02.084 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.345+0000 7eff5099e700 1 --2- 172.21.15.72:0/4044356141 >> v2:172.21.15.72:6800/347688983 conn(0x7eff38038ea0 0x7eff3803b360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T01:56:02.084 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.747+0000 7eff5099e700 1 -- 172.21.15.72:0/4044356141 >> v2:172.21.15.72:6800/347688983 conn(0x7eff38038ea0 msgr2=0x7eff3803b360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/347688983 2024-03-26T01:56:02.084 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:56.747+0000 7eff5099e700 1 --2- 172.21.15.72:0/4044356141 >> v2:172.21.15.72:6800/347688983 conn(0x7eff38038ea0 0x7eff3803b360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-26T01:56:02.084 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:57.548+0000 7eff5099e700 1 -- 172.21.15.72:0/4044356141 >> v2:172.21.15.72:6800/347688983 conn(0x7eff38038ea0 msgr2=0x7eff3803b360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/347688983 2024-03-26T01:56:02.084 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:57.548+0000 7eff5099e700 1 --2- 172.21.15.72:0/4044356141 >> v2:172.21.15.72:6800/347688983 conn(0x7eff38038ea0 0x7eff3803b360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-26T01:56:02.084 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:59.149+0000 7eff5099e700 1 -- 172.21.15.72:0/4044356141 >> v2:172.21.15.72:6800/347688983 conn(0x7eff38038ea0 msgr2=0x7eff3803b360 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/347688983 2024-03-26T01:56:02.084 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:55:59.149+0000 7eff5099e700 1 --2- 172.21.15.72:0/4044356141 >> v2:172.21.15.72:6800/347688983 conn(0x7eff38038ea0 0x7eff3803b360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-26T01:56:02.084 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:00.484+0000 7eff427fc700 1 -- 172.21.15.72:0/4044356141 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mgrmap(e 6) v1 ==== 45695+0+0 (secure 0 0 0) 0x7eff3c024b30 con 0x7eff4c104e00 2024-03-26T01:56:02.084 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:00.484+0000 7eff427fc700 1 -- 172.21.15.72:0/4044356141 >> v2:172.21.15.72:6800/347688983 conn(0x7eff38038ea0 msgr2=0x7eff3803b360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T01:56:02.084 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:00.484+0000 7eff427fc700 1 --2- 172.21.15.72:0/4044356141 >> v2:172.21.15.72:6800/347688983 conn(0x7eff38038ea0 0x7eff3803b360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:02.084 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:01.485+0000 7eff427fc700 1 -- 172.21.15.72:0/4044356141 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mgrmap(e 7) v1 ==== 45787+0+0 (secure 0 0 0) 0x7eff3c02d3f0 con 0x7eff4c104e00 2024-03-26T01:56:02.085 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:01.485+0000 7eff427fc700 1 --2- 172.21.15.72:0/4044356141 >> v2:172.21.15.72:6800/3912119602 conn(0x7eff3803c770 0x7eff3803eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:02.085 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:01.485+0000 7eff427fc700 1 -- 172.21.15.72:0/4044356141 --> v2:172.21.15.72:6800/3912119602 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7eff3803ba30 con 0x7eff3803c770 2024-03-26T01:56:02.085 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:01.485+0000 7eff5099e700 1 --2- 172.21.15.72:0/4044356141 >> v2:172.21.15.72:6800/3912119602 conn(0x7eff3803c770 0x7eff3803eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:02.085 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:01.486+0000 7eff5099e700 1 --2- 172.21.15.72:0/4044356141 >> v2:172.21.15.72:6800/3912119602 conn(0x7eff3803c770 0x7eff3803eb60 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7eff48003a10 tx=0x7eff480092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:02.085 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:01.487+0000 7eff427fc700 1 -- 172.21.15.72:0/4044356141 <== mgr.14120 v2:172.21.15.72:6800/3912119602 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7eff3803ba30 con 0x7eff3803c770 2024-03-26T01:56:02.085 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:01.493+0000 7eff521a1700 1 -- 172.21.15.72:0/4044356141 --> v2:172.21.15.72:6800/3912119602 -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7eff4c109220 con 0x7eff3803c770 2024-03-26T01:56:02.085 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:01.493+0000 7eff427fc700 1 -- 172.21.15.72:0/4044356141 <== mgr.14120 v2:172.21.15.72:6800/3912119602 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7eff4800a030 con 0x7eff3803c770 2024-03-26T01:56:02.085 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:01.494+0000 7eff521a1700 1 -- 172.21.15.72:0/4044356141 >> v2:172.21.15.72:6800/3912119602 conn(0x7eff3803c770 msgr2=0x7eff3803eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:02.085 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:01.494+0000 7eff521a1700 1 --2- 172.21.15.72:0/4044356141 >> v2:172.21.15.72:6800/3912119602 conn(0x7eff3803c770 0x7eff3803eb60 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7eff48003a10 tx=0x7eff480092b0 comp rx=0 tx=0).stop 2024-03-26T01:56:02.085 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:01.494+0000 7eff521a1700 1 -- 172.21.15.72:0/4044356141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff4c104e00 msgr2=0x7eff4c199cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:02.085 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:01.494+0000 7eff521a1700 1 --2- 172.21.15.72:0/4044356141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff4c104e00 0x7eff4c199cb0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7eff3c0046c0 tx=0x7eff3c011a50 comp rx=0 tx=0).stop 2024-03-26T01:56:02.085 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:01.494+0000 7eff521a1700 1 -- 172.21.15.72:0/4044356141 shutdown_connections 2024-03-26T01:56:02.085 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:01.494+0000 7eff521a1700 1 --2- 172.21.15.72:0/4044356141 >> v2:172.21.15.72:6800/3912119602 conn(0x7eff3803c770 0x7eff3803eb60 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:02.085 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:01.494+0000 7eff521a1700 1 --2- 172.21.15.72:0/4044356141 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7eff4c104e00 0x7eff4c199cb0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:02.086 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:01.494+0000 7eff521a1700 1 --2- 172.21.15.72:0/4044356141 >> v2:172.21.15.72:6800/347688983 conn(0x7eff38038ea0 0x7eff3803b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:02.086 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:01.494+0000 7eff521a1700 1 -- 172.21.15.72:0/4044356141 >> 172.21.15.72:0/4044356141 conn(0x7eff4c100380 msgr2=0x7eff4c101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:02.086 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:01.494+0000 7eff521a1700 1 -- 172.21.15.72:0/4044356141 shutdown_connections 2024-03-26T01:56:02.086 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:01.494+0000 7eff521a1700 1 -- 172.21.15.72:0/4044356141 wait complete. 2024-03-26T01:56:02.086 INFO:teuthology.orchestra.run.smithi072.stdout:mgr epoch 5 is available 2024-03-26T01:56:02.086 INFO:teuthology.orchestra.run.smithi072.stdout:Setting orchestrator backend to cephadm... 2024-03-26T01:56:02.718 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:02 smithi072 ceph-mon[30177]: [26/Mar/2024:01:56:01] ENGINE Bus STARTING 2024-03-26T01:56:02.718 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:02 smithi072 ceph-mon[30177]: [26/Mar/2024:01:56:01] ENGINE Serving on https://172.21.15.72:7150 2024-03-26T01:56:02.718 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:02 smithi072 ceph-mon[30177]: [26/Mar/2024:01:56:01] ENGINE Serving on http://172.21.15.72:8765 2024-03-26T01:56:02.718 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:02 smithi072 ceph-mon[30177]: [26/Mar/2024:01:56:01] ENGINE Bus STARTED 2024-03-26T01:56:02.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:02 smithi072 ceph-mon[30177]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T01:56:02.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:02 smithi072 ceph-mon[30177]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T01:56:03.266 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.510+0000 7ff6c509b700 1 Processor -- start 2024-03-26T01:56:03.266 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.510+0000 7ff6c509b700 1 -- start start 2024-03-26T01:56:03.267 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.510+0000 7ff6c509b700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6c0106b30 0x7ff6c0108f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:03.267 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.510+0000 7ff6c509b700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c0079720 con 0x7ff6c0106b30 2024-03-26T01:56:03.267 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.510+0000 7ff6bed9d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6c0106b30 0x7ff6c0108f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:03.267 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.511+0000 7ff6bed9d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6c0106b30 0x7ff6c0108f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37338/0 (socket says 172.21.15.72:37338) 2024-03-26T01:56:03.267 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.511+0000 7ff6bed9d700 1 -- 172.21.15.72:0/1406476960 learned_addr learned my addr 172.21.15.72:0/1406476960 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:03.267 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.511+0000 7ff6bed9d700 1 -- 172.21.15.72:0/1406476960 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c0109460 con 0x7ff6c0106b30 2024-03-26T01:56:03.267 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.511+0000 7ff6bed9d700 1 --2- 172.21.15.72:0/1406476960 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6c0106b30 0x7ff6c0108f20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff6b40098a0 tx=0x7ff6b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40ef27412638e4fe server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:03.267 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.511+0000 7ff6bdd9b700 1 -- 172.21.15.72:0/1406476960 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff6b4004160 con 0x7ff6c0106b30 2024-03-26T01:56:03.267 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.511+0000 7ff6bdd9b700 1 -- 172.21.15.72:0/1406476960 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7ff6b40042c0 con 0x7ff6c0106b30 2024-03-26T01:56:03.267 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.511+0000 7ff6bdd9b700 1 -- 172.21.15.72:0/1406476960 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff6b40119b0 con 0x7ff6c0106b30 2024-03-26T01:56:03.268 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.512+0000 7ff6c509b700 1 -- 172.21.15.72:0/1406476960 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6c0106b30 msgr2=0x7ff6c0108f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:03.268 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.512+0000 7ff6c509b700 1 --2- 172.21.15.72:0/1406476960 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6c0106b30 0x7ff6c0108f20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff6b40098a0 tx=0x7ff6b4009bb0 comp rx=0 tx=0).stop 2024-03-26T01:56:03.268 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.512+0000 7ff6c509b700 1 -- 172.21.15.72:0/1406476960 shutdown_connections 2024-03-26T01:56:03.268 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.512+0000 7ff6c509b700 1 --2- 172.21.15.72:0/1406476960 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6c0106b30 0x7ff6c0108f20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:03.268 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.512+0000 7ff6c509b700 1 -- 172.21.15.72:0/1406476960 >> 172.21.15.72:0/1406476960 conn(0x7ff6c01005a0 msgr2=0x7ff6c01029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:03.268 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.512+0000 7ff6c509b700 1 -- 172.21.15.72:0/1406476960 shutdown_connections 2024-03-26T01:56:03.268 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.512+0000 7ff6c509b700 1 -- 172.21.15.72:0/1406476960 wait complete. 2024-03-26T01:56:03.268 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.512+0000 7ff6c509b700 1 Processor -- start 2024-03-26T01:56:03.268 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.513+0000 7ff6c509b700 1 -- start start 2024-03-26T01:56:03.268 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.513+0000 7ff6c509b700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6c0106b30 0x7ff6c01063b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:03.268 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.513+0000 7ff6c509b700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c01068f0 con 0x7ff6c0106b30 2024-03-26T01:56:03.268 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.513+0000 7ff6bed9d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6c0106b30 0x7ff6c01063b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:03.268 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.513+0000 7ff6bed9d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6c0106b30 0x7ff6c01063b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37348/0 (socket says 172.21.15.72:37348) 2024-03-26T01:56:03.268 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.513+0000 7ff6bed9d700 1 -- 172.21.15.72:0/20358615 learned_addr learned my addr 172.21.15.72:0/20358615 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:03.269 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.513+0000 7ff6bed9d700 1 -- 172.21.15.72:0/20358615 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b4009550 con 0x7ff6c0106b30 2024-03-26T01:56:03.269 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.513+0000 7ff6bed9d700 1 --2- 172.21.15.72:0/20358615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6c0106b30 0x7ff6c01063b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4009130 tx=0x7ff6b4011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:03.269 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.513+0000 7ff6a7fff700 1 -- 172.21.15.72:0/20358615 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff6b4003aa0 con 0x7ff6c0106b30 2024-03-26T01:56:03.269 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.513+0000 7ff6a7fff700 1 -- 172.21.15.72:0/20358615 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7ff6b401a8c0 con 0x7ff6c0106b30 2024-03-26T01:56:03.269 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.514+0000 7ff6c509b700 1 -- 172.21.15.72:0/20358615 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6c0104ae0 con 0x7ff6c0106b30 2024-03-26T01:56:03.269 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.514+0000 7ff6a7fff700 1 -- 172.21.15.72:0/20358615 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff6b4019790 con 0x7ff6c0106b30 2024-03-26T01:56:03.269 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.514+0000 7ff6c509b700 1 -- 172.21.15.72:0/20358615 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6c0104f80 con 0x7ff6c0106b30 2024-03-26T01:56:03.270 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.514+0000 7ff6a7fff700 1 -- 172.21.15.72:0/20358615 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 7) v1 ==== 45787+0+0 (secure 0 0 0) 0x7ff6b4019a50 con 0x7ff6c0106b30 2024-03-26T01:56:03.270 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.514+0000 7ff6c509b700 1 -- 172.21.15.72:0/20358615 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6c010a260 con 0x7ff6c0106b30 2024-03-26T01:56:03.270 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.514+0000 7ff6a7fff700 1 --2- 172.21.15.72:0/20358615 >> v2:172.21.15.72:6800/3912119602 conn(0x7ff6a8038d80 0x7ff6a803b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:03.270 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.515+0000 7ff6a7fff700 1 -- 172.21.15.72:0/20358615 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7ff6b404cd70 con 0x7ff6c0106b30 2024-03-26T01:56:03.270 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.515+0000 7ff6be59c700 1 --2- 172.21.15.72:0/20358615 >> v2:172.21.15.72:6800/3912119602 conn(0x7ff6a8038d80 0x7ff6a803b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:03.270 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.515+0000 7ff6be59c700 1 --2- 172.21.15.72:0/20358615 >> v2:172.21.15.72:6800/3912119602 conn(0x7ff6a8038d80 0x7ff6a803b240 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0006fd0 tx=0x7ff6b0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:03.270 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.520+0000 7ff6a7fff700 1 -- 172.21.15.72:0/20358615 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff6b4018c20 con 0x7ff6c0106b30 2024-03-26T01:56:03.270 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.536+0000 7ff6a7fff700 1 -- 172.21.15.72:0/20358615 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mgrmap(e 8) v1 ==== 45893+0+0 (secure 0 0 0) 0x7ff6b4019d00 con 0x7ff6c0106b30 2024-03-26T01:56:03.270 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.732+0000 7ff6c509b700 1 -- 172.21.15.72:0/20358615 --> v2:172.21.15.72:6800/3912119602 -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6c01017e0 con 0x7ff6a8038d80 2024-03-26T01:56:03.270 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.739+0000 7ff6a7fff700 1 -- 172.21.15.72:0/20358615 <== mgr.14120 v2:172.21.15.72:6800/3912119602 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff6c01017e0 con 0x7ff6a8038d80 2024-03-26T01:56:03.270 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.741+0000 7ff6c509b700 1 -- 172.21.15.72:0/20358615 >> v2:172.21.15.72:6800/3912119602 conn(0x7ff6a8038d80 msgr2=0x7ff6a803b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:03.271 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.741+0000 7ff6c509b700 1 --2- 172.21.15.72:0/20358615 >> v2:172.21.15.72:6800/3912119602 conn(0x7ff6a8038d80 0x7ff6a803b240 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0006fd0 tx=0x7ff6b0006e40 comp rx=0 tx=0).stop 2024-03-26T01:56:03.271 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.741+0000 7ff6c509b700 1 -- 172.21.15.72:0/20358615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6c0106b30 msgr2=0x7ff6c01063b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:03.271 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.741+0000 7ff6c509b700 1 --2- 172.21.15.72:0/20358615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6c0106b30 0x7ff6c01063b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4009130 tx=0x7ff6b4011a50 comp rx=0 tx=0).stop 2024-03-26T01:56:03.271 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.741+0000 7ff6c509b700 1 -- 172.21.15.72:0/20358615 shutdown_connections 2024-03-26T01:56:03.271 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.741+0000 7ff6c509b700 1 --2- 172.21.15.72:0/20358615 >> v2:172.21.15.72:6800/3912119602 conn(0x7ff6a8038d80 0x7ff6a803b240 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:03.271 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.741+0000 7ff6c509b700 1 --2- 172.21.15.72:0/20358615 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff6c0106b30 0x7ff6c01063b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:03.271 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.741+0000 7ff6c509b700 1 -- 172.21.15.72:0/20358615 >> 172.21.15.72:0/20358615 conn(0x7ff6c01005a0 msgr2=0x7ff6c01010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:03.271 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.741+0000 7ff6c509b700 1 -- 172.21.15.72:0/20358615 shutdown_connections 2024-03-26T01:56:03.271 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:02.741+0000 7ff6c509b700 1 -- 172.21.15.72:0/20358615 wait complete. 2024-03-26T01:56:03.629 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:03 smithi072 ceph-mon[30177]: mgrmap e8: y(active, since 2s) 2024-03-26T01:56:03.629 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:03 smithi072 ceph-mon[30177]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:03.629 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:03 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' 2024-03-26T01:56:03.629 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:03 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:04.490 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-26T01:56:04.490 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.723+0000 7fbf2f985700 1 Processor -- start 2024-03-26T01:56:04.490 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.723+0000 7fbf2f985700 1 -- start start 2024-03-26T01:56:04.491 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.723+0000 7fbf2f985700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf2807bfc0 0x7fbf2807a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:04.491 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.723+0000 7fbf2f985700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2807a9f0 con 0x7fbf2807bfc0 2024-03-26T01:56:04.491 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.724+0000 7fbf2d721700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf2807bfc0 0x7fbf2807a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:04.491 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.724+0000 7fbf2d721700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf2807bfc0 0x7fbf2807a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37354/0 (socket says 172.21.15.72:37354) 2024-03-26T01:56:04.491 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.724+0000 7fbf2d721700 1 -- 172.21.15.72:0/1923705799 learned_addr learned my addr 172.21.15.72:0/1923705799 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:04.491 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.724+0000 7fbf2d721700 1 -- 172.21.15.72:0/1923705799 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf2807ab30 con 0x7fbf2807bfc0 2024-03-26T01:56:04.491 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.724+0000 7fbf2d721700 1 --2- 172.21.15.72:0/1923705799 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf2807bfc0 0x7fbf2807a420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbf240098a0 tx=0x7fbf24009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17a49eeb12e88ad5 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:04.491 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.724+0000 7fbf1ffff700 1 -- 172.21.15.72:0/1923705799 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf24004160 con 0x7fbf2807bfc0 2024-03-26T01:56:04.491 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.724+0000 7fbf1ffff700 1 -- 172.21.15.72:0/1923705799 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7fbf240042c0 con 0x7fbf2807bfc0 2024-03-26T01:56:04.491 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.724+0000 7fbf1ffff700 1 -- 172.21.15.72:0/1923705799 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf240119b0 con 0x7fbf2807bfc0 2024-03-26T01:56:04.491 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.725+0000 7fbf2f985700 1 -- 172.21.15.72:0/1923705799 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf2807bfc0 msgr2=0x7fbf2807a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:04.491 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.725+0000 7fbf2f985700 1 --2- 172.21.15.72:0/1923705799 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf2807bfc0 0x7fbf2807a420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fbf240098a0 tx=0x7fbf24009bb0 comp rx=0 tx=0).stop 2024-03-26T01:56:04.492 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.725+0000 7fbf2f985700 1 -- 172.21.15.72:0/1923705799 shutdown_connections 2024-03-26T01:56:04.492 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.725+0000 7fbf2f985700 1 --2- 172.21.15.72:0/1923705799 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf2807bfc0 0x7fbf2807a420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:04.492 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.725+0000 7fbf2f985700 1 -- 172.21.15.72:0/1923705799 >> 172.21.15.72:0/1923705799 conn(0x7fbf281016b0 msgr2=0x7fbf28103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:04.492 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.725+0000 7fbf2f985700 1 -- 172.21.15.72:0/1923705799 shutdown_connections 2024-03-26T01:56:04.492 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.725+0000 7fbf2f985700 1 -- 172.21.15.72:0/1923705799 wait complete. 2024-03-26T01:56:04.492 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.725+0000 7fbf2f985700 1 Processor -- start 2024-03-26T01:56:04.492 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.725+0000 7fbf2f985700 1 -- start start 2024-03-26T01:56:04.492 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.726+0000 7fbf2f985700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf2807bfc0 0x7fbf2819c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:04.492 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.726+0000 7fbf2f985700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2819c590 con 0x7fbf2807bfc0 2024-03-26T01:56:04.492 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.726+0000 7fbf2d721700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf2807bfc0 0x7fbf2819c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:04.492 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.726+0000 7fbf2d721700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf2807bfc0 0x7fbf2819c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37360/0 (socket says 172.21.15.72:37360) 2024-03-26T01:56:04.493 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.726+0000 7fbf2d721700 1 -- 172.21.15.72:0/1134337146 learned_addr learned my addr 172.21.15.72:0/1134337146 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:04.493 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.726+0000 7fbf2d721700 1 -- 172.21.15.72:0/1134337146 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf24009550 con 0x7fbf2807bfc0 2024-03-26T01:56:04.493 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.726+0000 7fbf2d721700 1 --2- 172.21.15.72:0/1134337146 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf2807bfc0 0x7fbf2819c050 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbf24006af0 tx=0x7fbf24011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:04.493 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.726+0000 7fbf1e7fc700 1 -- 172.21.15.72:0/1134337146 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf240039e0 con 0x7fbf2807bfc0 2024-03-26T01:56:04.493 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.726+0000 7fbf2f985700 1 -- 172.21.15.72:0/1134337146 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf2819c790 con 0x7fbf2807bfc0 2024-03-26T01:56:04.493 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.726+0000 7fbf1e7fc700 1 -- 172.21.15.72:0/1134337146 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7fbf24003b40 con 0x7fbf2807bfc0 2024-03-26T01:56:04.493 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.726+0000 7fbf1e7fc700 1 -- 172.21.15.72:0/1134337146 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbf24019720 con 0x7fbf2807bfc0 2024-03-26T01:56:04.493 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.726+0000 7fbf2f985700 1 -- 172.21.15.72:0/1134337146 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf2819cc30 con 0x7fbf2807bfc0 2024-03-26T01:56:04.493 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.727+0000 7fbf1e7fc700 1 -- 172.21.15.72:0/1134337146 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) v1 ==== 45893+0+0 (secure 0 0 0) 0x7fbf240199a0 con 0x7fbf2807bfc0 2024-03-26T01:56:04.493 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.727+0000 7fbf2f985700 1 -- 172.21.15.72:0/1134337146 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf2810c4a0 con 0x7fbf2807bfc0 2024-03-26T01:56:04.493 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.727+0000 7fbf1e7fc700 1 --2- 172.21.15.72:0/1134337146 >> v2:172.21.15.72:6800/3912119602 conn(0x7fbf14038ea0 0x7fbf1403b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:04.493 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.727+0000 7fbf1e7fc700 1 -- 172.21.15.72:0/1134337146 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7fbf2404d130 con 0x7fbf2807bfc0 2024-03-26T01:56:04.494 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.727+0000 7fbf2cf20700 1 --2- 172.21.15.72:0/1134337146 >> v2:172.21.15.72:6800/3912119602 conn(0x7fbf14038ea0 0x7fbf1403b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:04.494 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.728+0000 7fbf2cf20700 1 --2- 172.21.15.72:0/1134337146 >> v2:172.21.15.72:6800/3912119602 conn(0x7fbf14038ea0 0x7fbf1403b360 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbf18006fd0 tx=0x7fbf18006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:04.494 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.732+0000 7fbf1e7fc700 1 -- 172.21.15.72:0/1134337146 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fbf24010c50 con 0x7fbf2807bfc0 2024-03-26T01:56:04.494 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.924+0000 7fbf2f985700 1 -- 172.21.15.72:0/1134337146 --> v2:172.21.15.72:6800/3912119602 -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fbf281028d0 con 0x7fbf14038ea0 2024-03-26T01:56:04.494 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.925+0000 7fbf1e7fc700 1 -- 172.21.15.72:0/1134337146 <== mgr.14120 v2:172.21.15.72:6800/3912119602 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fbf281028d0 con 0x7fbf14038ea0 2024-03-26T01:56:04.494 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.926+0000 7fbf2f985700 1 -- 172.21.15.72:0/1134337146 >> v2:172.21.15.72:6800/3912119602 conn(0x7fbf14038ea0 msgr2=0x7fbf1403b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:04.494 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.926+0000 7fbf2f985700 1 --2- 172.21.15.72:0/1134337146 >> v2:172.21.15.72:6800/3912119602 conn(0x7fbf14038ea0 0x7fbf1403b360 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbf18006fd0 tx=0x7fbf18006e40 comp rx=0 tx=0).stop 2024-03-26T01:56:04.494 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.926+0000 7fbf2f985700 1 -- 172.21.15.72:0/1134337146 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf2807bfc0 msgr2=0x7fbf2819c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:04.494 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.926+0000 7fbf2f985700 1 --2- 172.21.15.72:0/1134337146 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf2807bfc0 0x7fbf2819c050 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fbf24006af0 tx=0x7fbf24011a50 comp rx=0 tx=0).stop 2024-03-26T01:56:04.494 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.927+0000 7fbf2f985700 1 -- 172.21.15.72:0/1134337146 shutdown_connections 2024-03-26T01:56:04.494 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.927+0000 7fbf2f985700 1 --2- 172.21.15.72:0/1134337146 >> v2:172.21.15.72:6800/3912119602 conn(0x7fbf14038ea0 0x7fbf1403b360 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:04.494 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.927+0000 7fbf2f985700 1 --2- 172.21.15.72:0/1134337146 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbf2807bfc0 0x7fbf2819c050 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:04.495 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.927+0000 7fbf2f985700 1 -- 172.21.15.72:0/1134337146 >> 172.21.15.72:0/1134337146 conn(0x7fbf281016b0 msgr2=0x7fbf281021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:04.495 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.927+0000 7fbf2f985700 1 -- 172.21.15.72:0/1134337146 shutdown_connections 2024-03-26T01:56:04.495 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:03.927+0000 7fbf2f985700 1 -- 172.21.15.72:0/1134337146 wait complete. 2024-03-26T01:56:04.495 INFO:teuthology.orchestra.run.smithi072.stdout:Generating ssh key... 2024-03-26T01:56:05.157 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:04 smithi072 ceph-mon[30177]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:05.468 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:05 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: Generating public/private rsa key pair. 2024-03-26T01:56:05.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:05 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: Your identification has been saved in /tmp/tmpz3p1xhzn/key. 2024-03-26T01:56:05.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:05 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: Your public key has been saved in /tmp/tmpz3p1xhzn/key.pub. 2024-03-26T01:56:05.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:05 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: The key fingerprint is: 2024-03-26T01:56:05.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:05 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: SHA256:MwEVFfooxY6ToEtb+gqLXsR4uHS7NhpAoA/01jQu91o ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T01:56:05.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:05 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: The key's randomart image is: 2024-03-26T01:56:05.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:05 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: +---[RSA 3072]----+ 2024-03-26T01:56:05.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:05 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: |.. o..ooo. | 2024-03-26T01:56:05.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:05 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: |+ . + .o . | 2024-03-26T01:56:05.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:05 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: |o. +.+ = | 2024-03-26T01:56:05.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:05 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: |.o=.o..= + | 2024-03-26T01:56:05.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:05 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: |.++=. =ES . | 2024-03-26T01:56:05.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:05 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: |o.==. oo o | 2024-03-26T01:56:05.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:05 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: |.o+o . | 2024-03-26T01:56:05.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:05 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: |.oo+. | 2024-03-26T01:56:05.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:05 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: |+o+oo | 2024-03-26T01:56:05.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:05 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: +----[SHA256]-----+ 2024-03-26T01:56:05.828 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.953+0000 7f8af59e1700 1 Processor -- start 2024-03-26T01:56:05.829 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.953+0000 7f8af59e1700 1 -- start start 2024-03-26T01:56:05.829 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.953+0000 7f8af59e1700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af01060f0 0x7f8af0106510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:05.829 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.953+0000 7f8af59e1700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af0106ae0 con 0x7f8af01060f0 2024-03-26T01:56:05.829 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.953+0000 7f8aeeffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af01060f0 0x7f8af0106510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:05.829 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.953+0000 7f8aeeffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af01060f0 0x7f8af0106510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37366/0 (socket says 172.21.15.72:37366) 2024-03-26T01:56:05.829 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.953+0000 7f8aeeffd700 1 -- 172.21.15.72:0/4012246447 learned_addr learned my addr 172.21.15.72:0/4012246447 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:05.829 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.953+0000 7f8aeeffd700 1 -- 172.21.15.72:0/4012246447 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af01072f0 con 0x7f8af01060f0 2024-03-26T01:56:05.829 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.953+0000 7f8aeeffd700 1 --2- 172.21.15.72:0/4012246447 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af01060f0 0x7f8af0106510 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8ae00098a0 tx=0x7f8ae0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20e02ae7b6c18270 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:05.829 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.954+0000 7f8aedffb700 1 -- 172.21.15.72:0/4012246447 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8ae0004160 con 0x7f8af01060f0 2024-03-26T01:56:05.829 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.954+0000 7f8aedffb700 1 -- 172.21.15.72:0/4012246447 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f8ae00042c0 con 0x7f8af01060f0 2024-03-26T01:56:05.830 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.954+0000 7f8aedffb700 1 -- 172.21.15.72:0/4012246447 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8ae00119b0 con 0x7f8af01060f0 2024-03-26T01:56:05.830 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.954+0000 7f8af59e1700 1 -- 172.21.15.72:0/4012246447 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af01060f0 msgr2=0x7f8af0106510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:05.830 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.954+0000 7f8af59e1700 1 --2- 172.21.15.72:0/4012246447 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af01060f0 0x7f8af0106510 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8ae00098a0 tx=0x7f8ae0009bb0 comp rx=0 tx=0).stop 2024-03-26T01:56:05.830 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.954+0000 7f8af59e1700 1 -- 172.21.15.72:0/4012246447 shutdown_connections 2024-03-26T01:56:05.830 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.954+0000 7f8af59e1700 1 --2- 172.21.15.72:0/4012246447 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af01060f0 0x7f8af0106510 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:05.830 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.954+0000 7f8af59e1700 1 -- 172.21.15.72:0/4012246447 >> 172.21.15.72:0/4012246447 conn(0x7f8af01016b0 msgr2=0x7f8af0103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:05.830 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.954+0000 7f8af59e1700 1 -- 172.21.15.72:0/4012246447 shutdown_connections 2024-03-26T01:56:05.830 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.954+0000 7f8af59e1700 1 -- 172.21.15.72:0/4012246447 wait complete. 2024-03-26T01:56:05.830 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.955+0000 7f8af59e1700 1 Processor -- start 2024-03-26T01:56:05.830 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.955+0000 7f8af59e1700 1 -- start start 2024-03-26T01:56:05.830 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.955+0000 7f8af59e1700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af01060f0 0x7f8af0199e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:05.830 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.955+0000 7f8af59e1700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af019a3b0 con 0x7f8af01060f0 2024-03-26T01:56:05.830 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.955+0000 7f8aeeffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af01060f0 0x7f8af0199e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:05.831 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.955+0000 7f8aeeffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af01060f0 0x7f8af0199e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37376/0 (socket says 172.21.15.72:37376) 2024-03-26T01:56:05.831 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.955+0000 7f8aeeffd700 1 -- 172.21.15.72:0/1407913417 learned_addr learned my addr 172.21.15.72:0/1407913417 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:05.831 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.956+0000 7f8aeeffd700 1 -- 172.21.15.72:0/1407913417 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae0009550 con 0x7f8af01060f0 2024-03-26T01:56:05.831 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.956+0000 7f8aeeffd700 1 --2- 172.21.15.72:0/1407913417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af01060f0 0x7f8af0199e70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8ae00046c0 tx=0x7f8ae0011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:05.831 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.956+0000 7f8af49df700 1 -- 172.21.15.72:0/1407913417 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8ae00039c0 con 0x7f8af01060f0 2024-03-26T01:56:05.831 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.956+0000 7f8af49df700 1 -- 172.21.15.72:0/1407913417 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f8ae0003b20 con 0x7f8af01060f0 2024-03-26T01:56:05.831 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.956+0000 7f8af59e1700 1 -- 172.21.15.72:0/1407913417 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af019a5b0 con 0x7f8af01060f0 2024-03-26T01:56:05.831 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.956+0000 7f8af49df700 1 -- 172.21.15.72:0/1407913417 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8ae001a780 con 0x7f8af01060f0 2024-03-26T01:56:05.831 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.956+0000 7f8af59e1700 1 -- 172.21.15.72:0/1407913417 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af019aa50 con 0x7f8af01060f0 2024-03-26T01:56:05.831 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.957+0000 7f8af49df700 1 -- 172.21.15.72:0/1407913417 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) v1 ==== 45893+0+0 (secure 0 0 0) 0x7f8ae001a9f0 con 0x7f8af01060f0 2024-03-26T01:56:05.831 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.957+0000 7f8af59e1700 1 -- 172.21.15.72:0/1407913417 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af00636b0 con 0x7f8af01060f0 2024-03-26T01:56:05.831 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.957+0000 7f8af49df700 1 --2- 172.21.15.72:0/1407913417 >> v2:172.21.15.72:6800/3912119602 conn(0x7f8ad803d260 0x7f8ad803f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:05.831 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.957+0000 7f8af49df700 1 -- 172.21.15.72:0/1407913417 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7f8ae004ccf0 con 0x7f8af01060f0 2024-03-26T01:56:05.831 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.957+0000 7f8aee7fc700 1 --2- 172.21.15.72:0/1407913417 >> v2:172.21.15.72:6800/3912119602 conn(0x7f8ad803d260 0x7f8ad803f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:05.832 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.957+0000 7f8aee7fc700 1 --2- 172.21.15.72:0/1407913417 >> v2:172.21.15.72:6800/3912119602 conn(0x7f8ad803d260 0x7f8ad803f720 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8ae4006fd0 tx=0x7f8ae4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:05.832 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:04.962+0000 7f8af49df700 1 -- 172.21.15.72:0/1407913417 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f8ae001aca0 con 0x7f8af01060f0 2024-03-26T01:56:05.832 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:05.156+0000 7f8af59e1700 1 -- 172.21.15.72:0/1407913417 --> v2:172.21.15.72:6800/3912119602 -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f8af0102930 con 0x7f8ad803d260 2024-03-26T01:56:05.832 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:05.265+0000 7f8af49df700 1 -- 172.21.15.72:0/1407913417 <== mgr.14120 v2:172.21.15.72:6800/3912119602 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8af0102930 con 0x7f8ad803d260 2024-03-26T01:56:05.832 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:05.267+0000 7f8af59e1700 1 -- 172.21.15.72:0/1407913417 >> v2:172.21.15.72:6800/3912119602 conn(0x7f8ad803d260 msgr2=0x7f8ad803f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:05.832 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:05.267+0000 7f8af59e1700 1 --2- 172.21.15.72:0/1407913417 >> v2:172.21.15.72:6800/3912119602 conn(0x7f8ad803d260 0x7f8ad803f720 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8ae4006fd0 tx=0x7f8ae4006e40 comp rx=0 tx=0).stop 2024-03-26T01:56:05.832 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:05.267+0000 7f8af59e1700 1 -- 172.21.15.72:0/1407913417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af01060f0 msgr2=0x7f8af0199e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:05.832 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:05.267+0000 7f8af59e1700 1 --2- 172.21.15.72:0/1407913417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af01060f0 0x7f8af0199e70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8ae00046c0 tx=0x7f8ae0011a50 comp rx=0 tx=0).stop 2024-03-26T01:56:05.832 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:05.267+0000 7f8af59e1700 1 -- 172.21.15.72:0/1407913417 shutdown_connections 2024-03-26T01:56:05.832 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:05.267+0000 7f8af59e1700 1 --2- 172.21.15.72:0/1407913417 >> v2:172.21.15.72:6800/3912119602 conn(0x7f8ad803d260 0x7f8ad803f720 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:05.832 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:05.267+0000 7f8af59e1700 1 --2- 172.21.15.72:0/1407913417 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8af01060f0 0x7f8af0199e70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:05.832 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:05.267+0000 7f8af59e1700 1 -- 172.21.15.72:0/1407913417 >> 172.21.15.72:0/1407913417 conn(0x7f8af01016b0 msgr2=0x7f8af0102220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:05.832 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:05.267+0000 7f8af59e1700 1 -- 172.21.15.72:0/1407913417 shutdown_connections 2024-03-26T01:56:05.832 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:05.267+0000 7f8af59e1700 1 -- 172.21.15.72:0/1407913417 wait complete. 2024-03-26T01:56:06.342 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:06 smithi072 ceph-mon[30177]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:06.342 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:06 smithi072 ceph-mon[30177]: Generating ssh key... 2024-03-26T01:56:06.342 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:06 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' 2024-03-26T01:56:06.342 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:06 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' 2024-03-26T01:56:07.086 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDafX/eCtR6yzLkLoyi6zM9XUNGF0VwQfVuDkeBWbODh4LOhGmzjkjDwh8crFLE0jViF8o5bZPNrj/Zls3czpgAV3LSNF1Oyckr0vKbH/Js6wXaTIp9b/EHarUwfO1s0h0ZoAAoSYjXJX/sNBERspEELeoRb5BbHrkHa/SdJA+L57skr9Hy6vXQ+yNjajzD3TlS6FYW3SwmWbXSl4euAEzQP2wd2NgobS3mwrhzyfI42L9YtPtnHkP/OObnBAdJtjAuJdHjq0xzUndFMVW1VBbpYLKHfBGZgzZie9lOL2/wLQGMRLRZf8KjwIZdD/Ui0yidfyppudiPdOyiT0f/KpGsLIRD9n5gQA99CKL786+bJqzFG8F2HhNZor5qx6cUY4jjJCCSeCEL1w/7hJJcV12/AKtG6/+mCMYjFE7kbj6/5ShSg5fY3RiG90sRTCRpGLPMsSmijkwO6p0gkswMFlUCIErUlAyW7LNowKDikihkuttvvEPfni2VzQqo5CZJns= ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T01:56:07.086 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.298+0000 7f53107f5700 1 Processor -- start 2024-03-26T01:56:07.086 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.298+0000 7f53107f5700 1 -- start start 2024-03-26T01:56:07.086 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.299+0000 7f53107f5700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f530807b730 0x7f5308079b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:07.086 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.299+0000 7f53107f5700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530807a160 con 0x7f530807b730 2024-03-26T01:56:07.086 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.299+0000 7f530e591700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f530807b730 0x7f5308079b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:07.086 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.299+0000 7f530e591700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f530807b730 0x7f5308079b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37386/0 (socket says 172.21.15.72:37386) 2024-03-26T01:56:07.086 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.299+0000 7f530e591700 1 -- 172.21.15.72:0/2601862833 learned_addr learned my addr 172.21.15.72:0/2601862833 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:07.086 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.299+0000 7f530e591700 1 -- 172.21.15.72:0/2601862833 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530807a2a0 con 0x7f530807b730 2024-03-26T01:56:07.087 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.299+0000 7f530e591700 1 --2- 172.21.15.72:0/2601862833 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f530807b730 0x7f5308079b90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f53040098a0 tx=0x7f5304009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7604d0549401373a server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:07.087 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.299+0000 7f530d58f700 1 -- 172.21.15.72:0/2601862833 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5304004160 con 0x7f530807b730 2024-03-26T01:56:07.087 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.299+0000 7f530d58f700 1 -- 172.21.15.72:0/2601862833 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f53040042c0 con 0x7f530807b730 2024-03-26T01:56:07.087 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.299+0000 7f530d58f700 1 -- 172.21.15.72:0/2601862833 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f53040119b0 con 0x7f530807b730 2024-03-26T01:56:07.087 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.300+0000 7f53107f5700 1 -- 172.21.15.72:0/2601862833 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f530807b730 msgr2=0x7f5308079b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:07.087 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.300+0000 7f53107f5700 1 --2- 172.21.15.72:0/2601862833 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f530807b730 0x7f5308079b90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f53040098a0 tx=0x7f5304009bb0 comp rx=0 tx=0).stop 2024-03-26T01:56:07.087 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.300+0000 7f53107f5700 1 -- 172.21.15.72:0/2601862833 shutdown_connections 2024-03-26T01:56:07.087 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.300+0000 7f53107f5700 1 --2- 172.21.15.72:0/2601862833 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f530807b730 0x7f5308079b90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:07.087 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.300+0000 7f53107f5700 1 -- 172.21.15.72:0/2601862833 >> 172.21.15.72:0/2601862833 conn(0x7f53081016a0 msgr2=0x7f5308103b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:07.087 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.300+0000 7f53107f5700 1 -- 172.21.15.72:0/2601862833 shutdown_connections 2024-03-26T01:56:07.087 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.300+0000 7f53107f5700 1 -- 172.21.15.72:0/2601862833 wait complete. 2024-03-26T01:56:07.087 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.301+0000 7f53107f5700 1 Processor -- start 2024-03-26T01:56:07.087 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.301+0000 7f53107f5700 1 -- start start 2024-03-26T01:56:07.087 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.301+0000 7f53107f5700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f530807b730 0x7f530819c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:07.088 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.301+0000 7f53107f5700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f530819c5e0 con 0x7f530807b730 2024-03-26T01:56:07.088 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.301+0000 7f530e591700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f530807b730 0x7f530819c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:07.088 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.301+0000 7f530e591700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f530807b730 0x7f530819c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37400/0 (socket says 172.21.15.72:37400) 2024-03-26T01:56:07.088 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.301+0000 7f530e591700 1 -- 172.21.15.72:0/4229320802 learned_addr learned my addr 172.21.15.72:0/4229320802 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:07.088 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.301+0000 7f530e591700 1 -- 172.21.15.72:0/4229320802 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5304009550 con 0x7f530807b730 2024-03-26T01:56:07.088 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.301+0000 7f530e591700 1 --2- 172.21.15.72:0/4229320802 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f530807b730 0x7f530819c0a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5304006af0 tx=0x7f5304011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:07.088 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.302+0000 7f52ff7fe700 1 -- 172.21.15.72:0/4229320802 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f53040039e0 con 0x7f530807b730 2024-03-26T01:56:07.088 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.302+0000 7f53107f5700 1 -- 172.21.15.72:0/4229320802 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f530819c7e0 con 0x7f530807b730 2024-03-26T01:56:07.088 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.302+0000 7f52ff7fe700 1 -- 172.21.15.72:0/4229320802 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f5304003b40 con 0x7f530807b730 2024-03-26T01:56:07.088 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.302+0000 7f52ff7fe700 1 -- 172.21.15.72:0/4229320802 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5304019720 con 0x7f530807b730 2024-03-26T01:56:07.088 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.302+0000 7f53107f5700 1 -- 172.21.15.72:0/4229320802 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f530819cc80 con 0x7f530807b730 2024-03-26T01:56:07.088 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.302+0000 7f52ff7fe700 1 -- 172.21.15.72:0/4229320802 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) v1 ==== 45893+0+0 (secure 0 0 0) 0x7f5304019a20 con 0x7f530807b730 2024-03-26T01:56:07.088 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.303+0000 7f52ff7fe700 1 --2- 172.21.15.72:0/4229320802 >> v2:172.21.15.72:6800/3912119602 conn(0x7f52f4038ea0 0x7f52f403b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:07.088 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.303+0000 7f52ff7fe700 1 -- 172.21.15.72:0/4229320802 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7f530404d0f0 con 0x7f530807b730 2024-03-26T01:56:07.089 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.303+0000 7f53107f5700 1 -- 172.21.15.72:0/4229320802 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f530810c4d0 con 0x7f530807b730 2024-03-26T01:56:07.089 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.303+0000 7f530dd90700 1 --2- 172.21.15.72:0/4229320802 >> v2:172.21.15.72:6800/3912119602 conn(0x7f52f4038ea0 0x7f52f403b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:07.089 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.303+0000 7f530dd90700 1 --2- 172.21.15.72:0/4229320802 >> v2:172.21.15.72:6800/3912119602 conn(0x7f52f4038ea0 0x7f52f403b360 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f52f8006fd0 tx=0x7f52f8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:07.089 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.308+0000 7f52ff7fe700 1 -- 172.21.15.72:0/4229320802 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5304023370 con 0x7f530807b730 2024-03-26T01:56:07.089 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.512+0000 7f53107f5700 1 -- 172.21.15.72:0/4229320802 --> v2:172.21.15.72:6800/3912119602 -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f53081028d0 con 0x7f52f4038ea0 2024-03-26T01:56:07.089 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.513+0000 7f52ff7fe700 1 -- 172.21.15.72:0/4229320802 <== mgr.14120 v2:172.21.15.72:6800/3912119602 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f53081028d0 con 0x7f52f4038ea0 2024-03-26T01:56:07.089 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.515+0000 7f53107f5700 1 -- 172.21.15.72:0/4229320802 >> v2:172.21.15.72:6800/3912119602 conn(0x7f52f4038ea0 msgr2=0x7f52f403b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:07.089 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.515+0000 7f53107f5700 1 --2- 172.21.15.72:0/4229320802 >> v2:172.21.15.72:6800/3912119602 conn(0x7f52f4038ea0 0x7f52f403b360 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f52f8006fd0 tx=0x7f52f8006e40 comp rx=0 tx=0).stop 2024-03-26T01:56:07.089 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.515+0000 7f53107f5700 1 -- 172.21.15.72:0/4229320802 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f530807b730 msgr2=0x7f530819c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:07.089 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.515+0000 7f53107f5700 1 --2- 172.21.15.72:0/4229320802 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f530807b730 0x7f530819c0a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5304006af0 tx=0x7f5304011a50 comp rx=0 tx=0).stop 2024-03-26T01:56:07.089 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.515+0000 7f53107f5700 1 -- 172.21.15.72:0/4229320802 shutdown_connections 2024-03-26T01:56:07.089 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.515+0000 7f53107f5700 1 --2- 172.21.15.72:0/4229320802 >> v2:172.21.15.72:6800/3912119602 conn(0x7f52f4038ea0 0x7f52f403b360 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:07.089 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.515+0000 7f53107f5700 1 --2- 172.21.15.72:0/4229320802 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f530807b730 0x7f530819c0a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:07.089 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.515+0000 7f53107f5700 1 -- 172.21.15.72:0/4229320802 >> 172.21.15.72:0/4229320802 conn(0x7f53081016a0 msgr2=0x7f53081021c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:07.089 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.515+0000 7f53107f5700 1 -- 172.21.15.72:0/4229320802 shutdown_connections 2024-03-26T01:56:07.090 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:06.515+0000 7f53107f5700 1 -- 172.21.15.72:0/4229320802 wait complete. 2024-03-26T01:56:07.090 INFO:teuthology.orchestra.run.smithi072.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-26T01:56:07.090 INFO:teuthology.orchestra.run.smithi072.stdout:Adding key to root@localhost authorized_keys... 2024-03-26T01:56:07.090 INFO:teuthology.orchestra.run.smithi072.stdout:Adding host smithi072... 2024-03-26T01:56:07.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:07 smithi072 ceph-mon[30177]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:08.416 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:08 smithi072 ceph-mon[30177]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi072", "addr": "172.21.15.72", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:09.718 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:09 smithi072 ceph-mon[30177]: Deploying cephadm binary to smithi072 2024-03-26T01:56:12.522 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout Added host 'smithi072' with addr '172.21.15.72' 2024-03-26T01:56:12.522 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.544+0000 7fe5f3cda700 1 Processor -- start 2024-03-26T01:56:12.522 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.544+0000 7fe5f3cda700 1 -- start start 2024-03-26T01:56:12.523 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.544+0000 7fe5f3cda700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5ec1060f0 0x7fe5ec106510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:12.523 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.544+0000 7fe5f3cda700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ec106ae0 con 0x7fe5ec1060f0 2024-03-26T01:56:12.523 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.545+0000 7fe5f1a76700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5ec1060f0 0x7fe5ec106510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:12.523 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.545+0000 7fe5f1a76700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5ec1060f0 0x7fe5ec106510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37404/0 (socket says 172.21.15.72:37404) 2024-03-26T01:56:12.523 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.545+0000 7fe5f1a76700 1 -- 172.21.15.72:0/4259367147 learned_addr learned my addr 172.21.15.72:0/4259367147 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:12.523 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.545+0000 7fe5f1a76700 1 -- 172.21.15.72:0/4259367147 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ec1072f0 con 0x7fe5ec1060f0 2024-03-26T01:56:12.523 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.545+0000 7fe5f1a76700 1 --2- 172.21.15.72:0/4259367147 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5ec1060f0 0x7fe5ec106510 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8009e80 tx=0x7fe5e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b667ff04f1920ebd server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:12.523 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.545+0000 7fe5f0a74700 1 -- 172.21.15.72:0/4259367147 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe5e800e070 con 0x7fe5ec1060f0 2024-03-26T01:56:12.523 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.545+0000 7fe5f0a74700 1 -- 172.21.15.72:0/4259367147 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7fe5e800b8c0 con 0x7fe5ec1060f0 2024-03-26T01:56:12.523 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.545+0000 7fe5f0a74700 1 -- 172.21.15.72:0/4259367147 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe5e80043e0 con 0x7fe5ec1060f0 2024-03-26T01:56:12.523 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.546+0000 7fe5f3cda700 1 -- 172.21.15.72:0/4259367147 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5ec1060f0 msgr2=0x7fe5ec106510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:12.523 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.546+0000 7fe5f3cda700 1 --2- 172.21.15.72:0/4259367147 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5ec1060f0 0x7fe5ec106510 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8009e80 tx=0x7fe5e800b330 comp rx=0 tx=0).stop 2024-03-26T01:56:12.523 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.546+0000 7fe5f3cda700 1 -- 172.21.15.72:0/4259367147 shutdown_connections 2024-03-26T01:56:12.524 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.546+0000 7fe5f3cda700 1 --2- 172.21.15.72:0/4259367147 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5ec1060f0 0x7fe5ec106510 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:12.524 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.546+0000 7fe5f3cda700 1 -- 172.21.15.72:0/4259367147 >> 172.21.15.72:0/4259367147 conn(0x7fe5ec1016b0 msgr2=0x7fe5ec103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:12.524 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.546+0000 7fe5f3cda700 1 -- 172.21.15.72:0/4259367147 shutdown_connections 2024-03-26T01:56:12.524 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.546+0000 7fe5f3cda700 1 -- 172.21.15.72:0/4259367147 wait complete. 2024-03-26T01:56:12.524 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.546+0000 7fe5f3cda700 1 Processor -- start 2024-03-26T01:56:12.524 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.546+0000 7fe5f3cda700 1 -- start start 2024-03-26T01:56:12.524 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.547+0000 7fe5f3cda700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5ec1060f0 0x7fe5ec199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:12.524 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.547+0000 7fe5f3cda700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ec19a350 con 0x7fe5ec1060f0 2024-03-26T01:56:12.524 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.547+0000 7fe5f1a76700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5ec1060f0 0x7fe5ec199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:12.524 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.547+0000 7fe5f1a76700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5ec1060f0 0x7fe5ec199e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:37420/0 (socket says 172.21.15.72:37420) 2024-03-26T01:56:12.524 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.547+0000 7fe5f1a76700 1 -- 172.21.15.72:0/3078317234 learned_addr learned my addr 172.21.15.72:0/3078317234 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:12.524 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.547+0000 7fe5f1a76700 1 -- 172.21.15.72:0/3078317234 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5e8009b30 con 0x7fe5ec1060f0 2024-03-26T01:56:12.524 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.547+0000 7fe5f1a76700 1 --2- 172.21.15.72:0/3078317234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5ec1060f0 0x7fe5ec199e10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8004970 tx=0x7fe5e800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:12.524 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.547+0000 7fe5e2ffd700 1 -- 172.21.15.72:0/3078317234 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe5e800e040 con 0x7fe5ec1060f0 2024-03-26T01:56:12.524 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.547+0000 7fe5e2ffd700 1 -- 172.21.15.72:0/3078317234 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7fe5e8003e60 con 0x7fe5ec1060f0 2024-03-26T01:56:12.525 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.547+0000 7fe5e2ffd700 1 -- 172.21.15.72:0/3078317234 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe5e8012790 con 0x7fe5ec1060f0 2024-03-26T01:56:12.525 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.547+0000 7fe5f3cda700 1 -- 172.21.15.72:0/3078317234 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5ec19a550 con 0x7fe5ec1060f0 2024-03-26T01:56:12.525 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.547+0000 7fe5f3cda700 1 -- 172.21.15.72:0/3078317234 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5ec19aa50 con 0x7fe5ec1060f0 2024-03-26T01:56:12.525 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.548+0000 7fe5e2ffd700 1 -- 172.21.15.72:0/3078317234 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) v1 ==== 45893+0+0 (secure 0 0 0) 0x7fe5e8012a00 con 0x7fe5ec1060f0 2024-03-26T01:56:12.525 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.548+0000 7fe5f3cda700 1 -- 172.21.15.72:0/3078317234 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5ec0636b0 con 0x7fe5ec1060f0 2024-03-26T01:56:12.525 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.548+0000 7fe5e2ffd700 1 --2- 172.21.15.72:0/3078317234 >> v2:172.21.15.72:6800/3912119602 conn(0x7fe5d8038ea0 0x7fe5d803b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:12.525 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.548+0000 7fe5e2ffd700 1 -- 172.21.15.72:0/3078317234 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7fe5e804d080 con 0x7fe5ec1060f0 2024-03-26T01:56:12.525 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.548+0000 7fe5f1275700 1 --2- 172.21.15.72:0/3078317234 >> v2:172.21.15.72:6800/3912119602 conn(0x7fe5d8038ea0 0x7fe5d803b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:12.525 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.549+0000 7fe5f1275700 1 --2- 172.21.15.72:0/3078317234 >> v2:172.21.15.72:6800/3912119602 conn(0x7fe5d8038ea0 0x7fe5d803b360 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc006fd0 tx=0x7fe5dc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:12.525 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.553+0000 7fe5e2ffd700 1 -- 172.21.15.72:0/3078317234 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe5e801bbf0 con 0x7fe5ec1060f0 2024-03-26T01:56:12.525 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:07.748+0000 7fe5f3cda700 1 -- 172.21.15.72:0/3078317234 --> v2:172.21.15.72:6800/3912119602 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi072", "addr": "172.21.15.72", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5ec102860 con 0x7fe5d8038ea0 2024-03-26T01:56:12.525 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:11.915+0000 7fe5e2ffd700 1 -- 172.21.15.72:0/3078317234 <== mgr.14120 v2:172.21.15.72:6800/3912119602 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fe5ec102860 con 0x7fe5d8038ea0 2024-03-26T01:56:12.526 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:11.917+0000 7fe5f3cda700 1 -- 172.21.15.72:0/3078317234 >> v2:172.21.15.72:6800/3912119602 conn(0x7fe5d8038ea0 msgr2=0x7fe5d803b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:12.526 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:11.917+0000 7fe5f3cda700 1 --2- 172.21.15.72:0/3078317234 >> v2:172.21.15.72:6800/3912119602 conn(0x7fe5d8038ea0 0x7fe5d803b360 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc006fd0 tx=0x7fe5dc006e40 comp rx=0 tx=0).stop 2024-03-26T01:56:12.526 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:11.917+0000 7fe5f3cda700 1 -- 172.21.15.72:0/3078317234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5ec1060f0 msgr2=0x7fe5ec199e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:12.526 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:11.917+0000 7fe5f3cda700 1 --2- 172.21.15.72:0/3078317234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5ec1060f0 0x7fe5ec199e10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe5e8004970 tx=0x7fe5e800bda0 comp rx=0 tx=0).stop 2024-03-26T01:56:12.526 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:11.917+0000 7fe5f3cda700 1 -- 172.21.15.72:0/3078317234 shutdown_connections 2024-03-26T01:56:12.526 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:11.917+0000 7fe5f3cda700 1 --2- 172.21.15.72:0/3078317234 >> v2:172.21.15.72:6800/3912119602 conn(0x7fe5d8038ea0 0x7fe5d803b360 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:12.526 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:11.917+0000 7fe5f3cda700 1 --2- 172.21.15.72:0/3078317234 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5ec1060f0 0x7fe5ec199e10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:12.526 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:11.917+0000 7fe5f3cda700 1 -- 172.21.15.72:0/3078317234 >> 172.21.15.72:0/3078317234 conn(0x7fe5ec1016b0 msgr2=0x7fe5ec102150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:12.526 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:11.917+0000 7fe5f3cda700 1 -- 172.21.15.72:0/3078317234 shutdown_connections 2024-03-26T01:56:12.526 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:11.917+0000 7fe5f3cda700 1 -- 172.21.15.72:0/3078317234 wait complete. 2024-03-26T01:56:12.527 INFO:teuthology.orchestra.run.smithi072.stdout:Deploying unmanaged mon service... 2024-03-26T01:56:13.117 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:12 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' 2024-03-26T01:56:13.117 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:12 smithi072 ceph-mon[30177]: Added host smithi072 2024-03-26T01:56:13.117 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:12 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:14.368 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-26T01:56:14.368 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.117+0000 7f4480fe2700 1 Processor -- start 2024-03-26T01:56:14.368 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.117+0000 7f4480fe2700 1 -- start start 2024-03-26T01:56:14.368 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.117+0000 7f4480fe2700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f447c104fc0 0x7f447c1053e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:14.368 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.117+0000 7f4480fe2700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447c1059b0 con 0x7f447c104fc0 2024-03-26T01:56:14.368 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.117+0000 7f447a59c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f447c104fc0 0x7f447c1053e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:14.368 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.117+0000 7f447a59c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f447c104fc0 0x7f447c1053e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53826/0 (socket says 172.21.15.72:53826) 2024-03-26T01:56:14.369 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.117+0000 7f447a59c700 1 -- 172.21.15.72:0/233253426 learned_addr learned my addr 172.21.15.72:0/233253426 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:14.369 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.118+0000 7f447a59c700 1 -- 172.21.15.72:0/233253426 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f447c1061c0 con 0x7f447c104fc0 2024-03-26T01:56:14.369 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.118+0000 7f447a59c700 1 --2- 172.21.15.72:0/233253426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f447c104fc0 0x7f447c1053e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f446c0098a0 tx=0x7f446c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e44ca5c93718d97 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:14.369 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.118+0000 7f447959a700 1 -- 172.21.15.72:0/233253426 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f446c004160 con 0x7f447c104fc0 2024-03-26T01:56:14.369 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.118+0000 7f447959a700 1 -- 172.21.15.72:0/233253426 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f446c0042c0 con 0x7f447c104fc0 2024-03-26T01:56:14.369 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.118+0000 7f447959a700 1 -- 172.21.15.72:0/233253426 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f446c0119b0 con 0x7f447c104fc0 2024-03-26T01:56:14.369 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.119+0000 7f4480fe2700 1 -- 172.21.15.72:0/233253426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f447c104fc0 msgr2=0x7f447c1053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:14.369 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.119+0000 7f4480fe2700 1 --2- 172.21.15.72:0/233253426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f447c104fc0 0x7f447c1053e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f446c0098a0 tx=0x7f446c009bb0 comp rx=0 tx=0).stop 2024-03-26T01:56:14.369 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.119+0000 7f4480fe2700 1 -- 172.21.15.72:0/233253426 shutdown_connections 2024-03-26T01:56:14.369 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.119+0000 7f4480fe2700 1 --2- 172.21.15.72:0/233253426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f447c104fc0 0x7f447c1053e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:14.369 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.119+0000 7f4480fe2700 1 -- 172.21.15.72:0/233253426 >> 172.21.15.72:0/233253426 conn(0x7f447c1005a0 msgr2=0x7f447c1029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:14.369 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.119+0000 7f4480fe2700 1 -- 172.21.15.72:0/233253426 shutdown_connections 2024-03-26T01:56:14.369 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.119+0000 7f4480fe2700 1 -- 172.21.15.72:0/233253426 wait complete. 2024-03-26T01:56:14.370 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.120+0000 7f4480fe2700 1 Processor -- start 2024-03-26T01:56:14.370 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.120+0000 7f4480fe2700 1 -- start start 2024-03-26T01:56:14.370 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.120+0000 7f4480fe2700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f447c104fc0 0x7f447c190070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:14.370 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.120+0000 7f4480fe2700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f447c1905b0 con 0x7f447c104fc0 2024-03-26T01:56:14.370 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.120+0000 7f447a59c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f447c104fc0 0x7f447c190070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:14.370 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.120+0000 7f447a59c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f447c104fc0 0x7f447c190070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53832/0 (socket says 172.21.15.72:53832) 2024-03-26T01:56:14.370 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.120+0000 7f447a59c700 1 -- 172.21.15.72:0/2781644236 learned_addr learned my addr 172.21.15.72:0/2781644236 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:14.370 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.120+0000 7f447a59c700 1 -- 172.21.15.72:0/2781644236 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f446c009550 con 0x7f447c104fc0 2024-03-26T01:56:14.370 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.120+0000 7f447a59c700 1 --2- 172.21.15.72:0/2781644236 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f447c104fc0 0x7f447c190070 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f446c006af0 tx=0x7f446c011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:14.370 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.120+0000 7f446b7fe700 1 -- 172.21.15.72:0/2781644236 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f446c0039e0 con 0x7f447c104fc0 2024-03-26T01:56:14.370 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.120+0000 7f446b7fe700 1 -- 172.21.15.72:0/2781644236 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1062+0+0 (secure 0 0 0) 0x7f446c003b40 con 0x7f447c104fc0 2024-03-26T01:56:14.370 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.120+0000 7f446b7fe700 1 -- 172.21.15.72:0/2781644236 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f446c019720 con 0x7f447c104fc0 2024-03-26T01:56:14.370 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.120+0000 7f4480fe2700 1 -- 172.21.15.72:0/2781644236 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f447c18e7a0 con 0x7f447c104fc0 2024-03-26T01:56:14.370 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.121+0000 7f4480fe2700 1 -- 172.21.15.72:0/2781644236 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f447c18ec40 con 0x7f447c104fc0 2024-03-26T01:56:14.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.121+0000 7f446b7fe700 1 -- 172.21.15.72:0/2781644236 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) v1 ==== 45893+0+0 (secure 0 0 0) 0x7f446c0199a0 con 0x7f447c104fc0 2024-03-26T01:56:14.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.121+0000 7f4480fe2700 1 -- 172.21.15.72:0/2781644236 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f447c18f650 con 0x7f447c104fc0 2024-03-26T01:56:14.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.121+0000 7f446b7fe700 1 --2- 172.21.15.72:0/2781644236 >> v2:172.21.15.72:6800/3912119602 conn(0x7f4464038ea0 0x7f446403b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:14.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.122+0000 7f446b7fe700 1 -- 172.21.15.72:0/2781644236 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7f446c04cf00 con 0x7f447c104fc0 2024-03-26T01:56:14.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.122+0000 7f4479d9b700 1 --2- 172.21.15.72:0/2781644236 >> v2:172.21.15.72:6800/3912119602 conn(0x7f4464038ea0 0x7f446403b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:14.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.122+0000 7f4479d9b700 1 --2- 172.21.15.72:0/2781644236 >> v2:172.21.15.72:6800/3912119602 conn(0x7f4464038ea0 0x7f446403b360 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4470006fd0 tx=0x7f4470006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:14.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.127+0000 7f446b7fe700 1 -- 172.21.15.72:0/2781644236 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f446c018d80 con 0x7f447c104fc0 2024-03-26T01:56:14.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.346+0000 7f4480fe2700 1 -- 172.21.15.72:0/2781644236 --> v2:172.21.15.72:6800/3912119602 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f447c18ee80 con 0x7f4464038ea0 2024-03-26T01:56:14.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.354+0000 7f446b7fe700 1 -- 172.21.15.72:0/2781644236 <== mgr.14120 v2:172.21.15.72:6800/3912119602 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f447c18ee80 con 0x7f4464038ea0 2024-03-26T01:56:14.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.356+0000 7f4480fe2700 1 -- 172.21.15.72:0/2781644236 >> v2:172.21.15.72:6800/3912119602 conn(0x7f4464038ea0 msgr2=0x7f446403b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:14.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.356+0000 7f4480fe2700 1 --2- 172.21.15.72:0/2781644236 >> v2:172.21.15.72:6800/3912119602 conn(0x7f4464038ea0 0x7f446403b360 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4470006fd0 tx=0x7f4470006e40 comp rx=0 tx=0).stop 2024-03-26T01:56:14.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.356+0000 7f4480fe2700 1 -- 172.21.15.72:0/2781644236 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f447c104fc0 msgr2=0x7f447c190070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:14.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.356+0000 7f4480fe2700 1 --2- 172.21.15.72:0/2781644236 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f447c104fc0 0x7f447c190070 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f446c006af0 tx=0x7f446c011a50 comp rx=0 tx=0).stop 2024-03-26T01:56:14.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.356+0000 7f4480fe2700 1 -- 172.21.15.72:0/2781644236 shutdown_connections 2024-03-26T01:56:14.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.356+0000 7f4480fe2700 1 --2- 172.21.15.72:0/2781644236 >> v2:172.21.15.72:6800/3912119602 conn(0x7f4464038ea0 0x7f446403b360 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:14.372 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.356+0000 7f4480fe2700 1 --2- 172.21.15.72:0/2781644236 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f447c104fc0 0x7f447c190070 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:14.372 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.356+0000 7f4480fe2700 1 -- 172.21.15.72:0/2781644236 >> 172.21.15.72:0/2781644236 conn(0x7f447c1005a0 msgr2=0x7f447c101530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:14.372 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.356+0000 7f4480fe2700 1 -- 172.21.15.72:0/2781644236 shutdown_connections 2024-03-26T01:56:14.372 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:13.356+0000 7f4480fe2700 1 -- 172.21.15.72:0/2781644236 wait complete. 2024-03-26T01:56:14.372 INFO:teuthology.orchestra.run.smithi072.stdout:Deploying unmanaged mgr service... 2024-03-26T01:56:14.706 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:14 smithi072 ceph-mon[30177]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:14.706 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:14 smithi072 ceph-mon[30177]: Saving service mon spec with placement count:5 2024-03-26T01:56:14.706 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:14 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' 2024-03-26T01:56:15.979 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-26T01:56:15.980 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.185+0000 7ff86c901700 1 Processor -- start 2024-03-26T01:56:15.980 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.185+0000 7ff86c901700 1 -- start start 2024-03-26T01:56:15.980 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.186+0000 7ff86c901700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff868104fc0 0x7ff8681053e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:15.980 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.186+0000 7ff86c901700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8681059b0 con 0x7ff868104fc0 2024-03-26T01:56:15.980 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.186+0000 7ff86659c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff868104fc0 0x7ff8681053e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:15.980 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.186+0000 7ff86659c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff868104fc0 0x7ff8681053e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53836/0 (socket says 172.21.15.72:53836) 2024-03-26T01:56:15.980 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.186+0000 7ff86659c700 1 -- 172.21.15.72:0/3009260742 learned_addr learned my addr 172.21.15.72:0/3009260742 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:15.980 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.186+0000 7ff86659c700 1 -- 172.21.15.72:0/3009260742 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8681061c0 con 0x7ff868104fc0 2024-03-26T01:56:15.980 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.186+0000 7ff86659c700 1 --2- 172.21.15.72:0/3009260742 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff868104fc0 0x7ff8681053e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff8580098a0 tx=0x7ff858009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8699dc38a42ce771 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:15.980 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.186+0000 7ff86559a700 1 -- 172.21.15.72:0/3009260742 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff858004160 con 0x7ff868104fc0 2024-03-26T01:56:15.980 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.186+0000 7ff86559a700 1 -- 172.21.15.72:0/3009260742 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7ff8580042c0 con 0x7ff868104fc0 2024-03-26T01:56:15.980 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.186+0000 7ff86559a700 1 -- 172.21.15.72:0/3009260742 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff858011960 con 0x7ff868104fc0 2024-03-26T01:56:15.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.187+0000 7ff86c901700 1 -- 172.21.15.72:0/3009260742 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff868104fc0 msgr2=0x7ff8681053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:15.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.187+0000 7ff86c901700 1 --2- 172.21.15.72:0/3009260742 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff868104fc0 0x7ff8681053e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff8580098a0 tx=0x7ff858009bb0 comp rx=0 tx=0).stop 2024-03-26T01:56:15.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.187+0000 7ff86c901700 1 -- 172.21.15.72:0/3009260742 shutdown_connections 2024-03-26T01:56:15.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.187+0000 7ff86c901700 1 --2- 172.21.15.72:0/3009260742 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff868104fc0 0x7ff8681053e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:15.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.187+0000 7ff86c901700 1 -- 172.21.15.72:0/3009260742 >> 172.21.15.72:0/3009260742 conn(0x7ff8681005a0 msgr2=0x7ff8681029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:15.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.187+0000 7ff86c901700 1 -- 172.21.15.72:0/3009260742 shutdown_connections 2024-03-26T01:56:15.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.187+0000 7ff86c901700 1 -- 172.21.15.72:0/3009260742 wait complete. 2024-03-26T01:56:15.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.188+0000 7ff86c901700 1 Processor -- start 2024-03-26T01:56:15.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.188+0000 7ff86c901700 1 -- start start 2024-03-26T01:56:15.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.188+0000 7ff86c901700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff868104fc0 0x7ff868079320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:15.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.188+0000 7ff86c901700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86819cba0 con 0x7ff868104fc0 2024-03-26T01:56:15.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.189+0000 7ff86659c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff868104fc0 0x7ff868079320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:15.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.189+0000 7ff86659c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff868104fc0 0x7ff868079320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53840/0 (socket says 172.21.15.72:53840) 2024-03-26T01:56:15.981 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.189+0000 7ff86659c700 1 -- 172.21.15.72:0/942452948 learned_addr learned my addr 172.21.15.72:0/942452948 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:15.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.189+0000 7ff86659c700 1 -- 172.21.15.72:0/942452948 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff858009550 con 0x7ff868104fc0 2024-03-26T01:56:15.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.189+0000 7ff86659c700 1 --2- 172.21.15.72:0/942452948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff868104fc0 0x7ff868079320 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff858006af0 tx=0x7ff8580119e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:15.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.189+0000 7ff8577fe700 1 -- 172.21.15.72:0/942452948 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff8580039b0 con 0x7ff868104fc0 2024-03-26T01:56:15.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.189+0000 7ff8577fe700 1 -- 172.21.15.72:0/942452948 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7ff858003b10 con 0x7ff868104fc0 2024-03-26T01:56:15.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.189+0000 7ff8577fe700 1 -- 172.21.15.72:0/942452948 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff858019670 con 0x7ff868104fc0 2024-03-26T01:56:15.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.189+0000 7ff86c901700 1 -- 172.21.15.72:0/942452948 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff86819cce0 con 0x7ff868104fc0 2024-03-26T01:56:15.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.189+0000 7ff86c901700 1 -- 172.21.15.72:0/942452948 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff86819d0c0 con 0x7ff868104fc0 2024-03-26T01:56:15.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.190+0000 7ff8577fe700 1 -- 172.21.15.72:0/942452948 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) v1 ==== 45893+0+0 (secure 0 0 0) 0x7ff8580198e0 con 0x7ff868104fc0 2024-03-26T01:56:15.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.190+0000 7ff86c901700 1 -- 172.21.15.72:0/942452948 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8681093f0 con 0x7ff868104fc0 2024-03-26T01:56:15.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.190+0000 7ff8577fe700 1 --2- 172.21.15.72:0/942452948 >> v2:172.21.15.72:6800/3912119602 conn(0x7ff850038ea0 0x7ff85003b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:15.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.190+0000 7ff8577fe700 1 -- 172.21.15.72:0/942452948 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7ff85804d1a0 con 0x7ff868104fc0 2024-03-26T01:56:15.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.191+0000 7ff865d9b700 1 --2- 172.21.15.72:0/942452948 >> v2:172.21.15.72:6800/3912119602 conn(0x7ff850038ea0 0x7ff85003b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:15.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.191+0000 7ff865d9b700 1 --2- 172.21.15.72:0/942452948 >> v2:172.21.15.72:6800/3912119602 conn(0x7ff850038ea0 0x7ff85003b360 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff85c006fd0 tx=0x7ff85c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:15.982 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.197+0000 7ff8577fe700 1 -- 172.21.15.72:0/942452948 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff858023e00 con 0x7ff868104fc0 2024-03-26T01:56:15.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.410+0000 7ff86c901700 1 -- 172.21.15.72:0/942452948 --> v2:172.21.15.72:6800/3912119602 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7ff8681018b0 con 0x7ff850038ea0 2024-03-26T01:56:15.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.417+0000 7ff8577fe700 1 -- 172.21.15.72:0/942452948 <== mgr.14120 v2:172.21.15.72:6800/3912119602 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7ff8681018b0 con 0x7ff850038ea0 2024-03-26T01:56:15.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.419+0000 7ff86c901700 1 -- 172.21.15.72:0/942452948 >> v2:172.21.15.72:6800/3912119602 conn(0x7ff850038ea0 msgr2=0x7ff85003b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:15.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.419+0000 7ff86c901700 1 --2- 172.21.15.72:0/942452948 >> v2:172.21.15.72:6800/3912119602 conn(0x7ff850038ea0 0x7ff85003b360 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff85c006fd0 tx=0x7ff85c006e40 comp rx=0 tx=0).stop 2024-03-26T01:56:15.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.419+0000 7ff86c901700 1 -- 172.21.15.72:0/942452948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff868104fc0 msgr2=0x7ff868079320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:15.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.419+0000 7ff86c901700 1 --2- 172.21.15.72:0/942452948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff868104fc0 0x7ff868079320 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff858006af0 tx=0x7ff8580119e0 comp rx=0 tx=0).stop 2024-03-26T01:56:15.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.419+0000 7ff86c901700 1 -- 172.21.15.72:0/942452948 shutdown_connections 2024-03-26T01:56:15.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.419+0000 7ff86c901700 1 --2- 172.21.15.72:0/942452948 >> v2:172.21.15.72:6800/3912119602 conn(0x7ff850038ea0 0x7ff85003b360 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:15.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.419+0000 7ff86c901700 1 --2- 172.21.15.72:0/942452948 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff868104fc0 0x7ff868079320 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:15.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.419+0000 7ff86c901700 1 -- 172.21.15.72:0/942452948 >> 172.21.15.72:0/942452948 conn(0x7ff8681005a0 msgr2=0x7ff8681011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:15.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.420+0000 7ff86c901700 1 -- 172.21.15.72:0/942452948 shutdown_connections 2024-03-26T01:56:15.983 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:15.420+0000 7ff86c901700 1 -- 172.21.15.72:0/942452948 wait complete. 2024-03-26T01:56:16.090 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:15 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' 2024-03-26T01:56:16.090 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:15 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' 2024-03-26T01:56:16.090 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:15 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' 2024-03-26T01:56:16.968 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:16 smithi072 ceph-mon[30177]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:16.968 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:16 smithi072 ceph-mon[30177]: Saving service mgr spec with placement count:2 2024-03-26T01:56:16.968 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:16 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3135820350' entity='client.admin' 2024-03-26T01:56:17.340 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.413+0000 7f3f7b177700 1 Processor -- start 2024-03-26T01:56:17.341 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.413+0000 7f3f7b177700 1 -- start start 2024-03-26T01:56:17.341 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.414+0000 7f3f7b177700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f74106ba0 0x7f3f74108f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:17.341 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.414+0000 7f3f7b177700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f74079f70 con 0x7f3f74106ba0 2024-03-26T01:56:17.341 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.414+0000 7f3f78f13700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f74106ba0 0x7f3f74108f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:17.341 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.414+0000 7f3f78f13700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f74106ba0 0x7f3f74108f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53842/0 (socket says 172.21.15.72:53842) 2024-03-26T01:56:17.341 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.414+0000 7f3f78f13700 1 -- 172.21.15.72:0/1750302682 learned_addr learned my addr 172.21.15.72:0/1750302682 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:17.341 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.414+0000 7f3f78f13700 1 -- 172.21.15.72:0/1750302682 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f741094d0 con 0x7f3f74106ba0 2024-03-26T01:56:17.341 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.414+0000 7f3f78f13700 1 --2- 172.21.15.72:0/1750302682 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f74106ba0 0x7f3f74108f90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3f680098a0 tx=0x7f3f68009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b289986004c1f013 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:17.341 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.414+0000 7f3f737fe700 1 -- 172.21.15.72:0/1750302682 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f68004160 con 0x7f3f74106ba0 2024-03-26T01:56:17.341 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.414+0000 7f3f737fe700 1 -- 172.21.15.72:0/1750302682 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f3f680042c0 con 0x7f3f74106ba0 2024-03-26T01:56:17.342 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.414+0000 7f3f737fe700 1 -- 172.21.15.72:0/1750302682 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f68011960 con 0x7f3f74106ba0 2024-03-26T01:56:17.342 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.415+0000 7f3f7b177700 1 -- 172.21.15.72:0/1750302682 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f74106ba0 msgr2=0x7f3f74108f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:17.342 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.415+0000 7f3f7b177700 1 --2- 172.21.15.72:0/1750302682 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f74106ba0 0x7f3f74108f90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3f680098a0 tx=0x7f3f68009bb0 comp rx=0 tx=0).stop 2024-03-26T01:56:17.342 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.415+0000 7f3f7b177700 1 -- 172.21.15.72:0/1750302682 shutdown_connections 2024-03-26T01:56:17.342 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.415+0000 7f3f7b177700 1 --2- 172.21.15.72:0/1750302682 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f74106ba0 0x7f3f74108f90 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:17.342 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.415+0000 7f3f7b177700 1 -- 172.21.15.72:0/1750302682 >> 172.21.15.72:0/1750302682 conn(0x7f3f741005b0 msgr2=0x7f3f741029d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:17.342 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.415+0000 7f3f7b177700 1 -- 172.21.15.72:0/1750302682 shutdown_connections 2024-03-26T01:56:17.342 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.415+0000 7f3f7b177700 1 -- 172.21.15.72:0/1750302682 wait complete. 2024-03-26T01:56:17.342 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.415+0000 7f3f7b177700 1 Processor -- start 2024-03-26T01:56:17.342 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.416+0000 7f3f7b177700 1 -- start start 2024-03-26T01:56:17.342 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.416+0000 7f3f7b177700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f74106ba0 0x7f3f74199e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:17.342 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.416+0000 7f3f7b177700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7419a380 con 0x7f3f74106ba0 2024-03-26T01:56:17.343 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.416+0000 7f3f78f13700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f74106ba0 0x7f3f74199e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:17.343 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.416+0000 7f3f78f13700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f74106ba0 0x7f3f74199e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53850/0 (socket says 172.21.15.72:53850) 2024-03-26T01:56:17.343 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.416+0000 7f3f78f13700 1 -- 172.21.15.72:0/3135820350 learned_addr learned my addr 172.21.15.72:0/3135820350 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:17.343 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.416+0000 7f3f78f13700 1 -- 172.21.15.72:0/3135820350 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f68009550 con 0x7f3f74106ba0 2024-03-26T01:56:17.343 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.416+0000 7f3f78f13700 1 --2- 172.21.15.72:0/3135820350 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f74106ba0 0x7f3f74199e40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3f68006af0 tx=0x7f3f680119e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:17.343 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.416+0000 7f3f71ffb700 1 -- 172.21.15.72:0/3135820350 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f680039b0 con 0x7f3f74106ba0 2024-03-26T01:56:17.343 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.416+0000 7f3f71ffb700 1 -- 172.21.15.72:0/3135820350 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f3f68003b10 con 0x7f3f74106ba0 2024-03-26T01:56:17.343 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.416+0000 7f3f7b177700 1 -- 172.21.15.72:0/3135820350 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f7419a580 con 0x7f3f74106ba0 2024-03-26T01:56:17.343 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.416+0000 7f3f71ffb700 1 -- 172.21.15.72:0/3135820350 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3f68019680 con 0x7f3f74106ba0 2024-03-26T01:56:17.343 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.417+0000 7f3f7b177700 1 -- 172.21.15.72:0/3135820350 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f7419aa20 con 0x7f3f74106ba0 2024-03-26T01:56:17.343 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.417+0000 7f3f71ffb700 1 -- 172.21.15.72:0/3135820350 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) v1 ==== 45893+0+0 (secure 0 0 0) 0x7f3f68019940 con 0x7f3f74106ba0 2024-03-26T01:56:17.343 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.418+0000 7f3f7b177700 1 -- 172.21.15.72:0/3135820350 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f7410a2b0 con 0x7f3f74106ba0 2024-03-26T01:56:17.344 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.418+0000 7f3f71ffb700 1 --2- 172.21.15.72:0/3135820350 >> v2:172.21.15.72:6800/3912119602 conn(0x7f3f64038e50 0x7f3f6403b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:17.344 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.418+0000 7f3f71ffb700 1 -- 172.21.15.72:0/3135820350 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7f3f6804d110 con 0x7f3f74106ba0 2024-03-26T01:56:17.344 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.418+0000 7f3f73fff700 1 --2- 172.21.15.72:0/3135820350 >> v2:172.21.15.72:6800/3912119602 conn(0x7f3f64038e50 0x7f3f6403b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:17.344 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.418+0000 7f3f73fff700 1 --2- 172.21.15.72:0/3135820350 >> v2:172.21.15.72:6800/3912119602 conn(0x7f3f64038e50 0x7f3f6403b310 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3f60006fd0 tx=0x7f3f60006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:17.344 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.423+0000 7f3f71ffb700 1 -- 172.21.15.72:0/3135820350 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3f68018d70 con 0x7f3f74106ba0 2024-03-26T01:56:17.344 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.636+0000 7f3f7b177700 1 -- 172.21.15.72:0/3135820350 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f3f7419d300 con 0x7f3f74106ba0 2024-03-26T01:56:17.344 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.639+0000 7f3f71ffb700 1 -- 172.21.15.72:0/3135820350 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f3f68018b90 con 0x7f3f74106ba0 2024-03-26T01:56:17.344 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.641+0000 7f3f7b177700 1 -- 172.21.15.72:0/3135820350 >> v2:172.21.15.72:6800/3912119602 conn(0x7f3f64038e50 msgr2=0x7f3f6403b310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:17.344 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.641+0000 7f3f7b177700 1 --2- 172.21.15.72:0/3135820350 >> v2:172.21.15.72:6800/3912119602 conn(0x7f3f64038e50 0x7f3f6403b310 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3f60006fd0 tx=0x7f3f60006e40 comp rx=0 tx=0).stop 2024-03-26T01:56:17.344 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.641+0000 7f3f7b177700 1 -- 172.21.15.72:0/3135820350 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f74106ba0 msgr2=0x7f3f74199e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:17.344 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.641+0000 7f3f7b177700 1 --2- 172.21.15.72:0/3135820350 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f74106ba0 0x7f3f74199e40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3f68006af0 tx=0x7f3f680119e0 comp rx=0 tx=0).stop 2024-03-26T01:56:17.345 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.641+0000 7f3f7b177700 1 -- 172.21.15.72:0/3135820350 shutdown_connections 2024-03-26T01:56:17.345 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.641+0000 7f3f7b177700 1 --2- 172.21.15.72:0/3135820350 >> v2:172.21.15.72:6800/3912119602 conn(0x7f3f64038e50 0x7f3f6403b310 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:17.345 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.641+0000 7f3f7b177700 1 --2- 172.21.15.72:0/3135820350 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3f74106ba0 0x7f3f74199e40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:17.345 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.641+0000 7f3f7b177700 1 -- 172.21.15.72:0/3135820350 >> 172.21.15.72:0/3135820350 conn(0x7f3f741005b0 msgr2=0x7f3f74101080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:17.345 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.641+0000 7f3f7b177700 1 -- 172.21.15.72:0/3135820350 shutdown_connections 2024-03-26T01:56:17.345 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:16.641+0000 7f3f7b177700 1 -- 172.21.15.72:0/3135820350 wait complete. 2024-03-26T01:56:18.759 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.900+0000 7fc3f2a4a700 1 Processor -- start 2024-03-26T01:56:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.900+0000 7fc3f2a4a700 1 -- start start 2024-03-26T01:56:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.901+0000 7fc3f2a4a700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec1060e0 0x7fc3ec106500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.901+0000 7fc3f2a4a700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec106ad0 con 0x7fc3ec1060e0 2024-03-26T01:56:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.901+0000 7fc3ebfff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec1060e0 0x7fc3ec106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.901+0000 7fc3ebfff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec1060e0 0x7fc3ec106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53860/0 (socket says 172.21.15.72:53860) 2024-03-26T01:56:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.901+0000 7fc3ebfff700 1 -- 172.21.15.72:0/2039757490 learned_addr learned my addr 172.21.15.72:0/2039757490 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.901+0000 7fc3ebfff700 1 -- 172.21.15.72:0/2039757490 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ec1072e0 con 0x7fc3ec1060e0 2024-03-26T01:56:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.901+0000 7fc3ebfff700 1 --2- 172.21.15.72:0/2039757490 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec1060e0 0x7fc3ec106500 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0098a0 tx=0x7fc3dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=841c5b88b30ba203 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.901+0000 7fc3eb7fe700 1 -- 172.21.15.72:0/2039757490 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc3dc004160 con 0x7fc3ec1060e0 2024-03-26T01:56:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.901+0000 7fc3eb7fe700 1 -- 172.21.15.72:0/2039757490 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fc3dc0042c0 con 0x7fc3ec1060e0 2024-03-26T01:56:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.901+0000 7fc3eb7fe700 1 -- 172.21.15.72:0/2039757490 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc3dc011960 con 0x7fc3ec1060e0 2024-03-26T01:56:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.902+0000 7fc3f2a4a700 1 -- 172.21.15.72:0/2039757490 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec1060e0 msgr2=0x7fc3ec106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.902+0000 7fc3f2a4a700 1 --2- 172.21.15.72:0/2039757490 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec1060e0 0x7fc3ec106500 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc0098a0 tx=0x7fc3dc009bb0 comp rx=0 tx=0).stop 2024-03-26T01:56:18.760 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.902+0000 7fc3f2a4a700 1 -- 172.21.15.72:0/2039757490 shutdown_connections 2024-03-26T01:56:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.902+0000 7fc3f2a4a700 1 --2- 172.21.15.72:0/2039757490 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec1060e0 0x7fc3ec106500 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.902+0000 7fc3f2a4a700 1 -- 172.21.15.72:0/2039757490 >> 172.21.15.72:0/2039757490 conn(0x7fc3ec1016a0 msgr2=0x7fc3ec103ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.902+0000 7fc3f2a4a700 1 -- 172.21.15.72:0/2039757490 shutdown_connections 2024-03-26T01:56:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.902+0000 7fc3f2a4a700 1 -- 172.21.15.72:0/2039757490 wait complete. 2024-03-26T01:56:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.903+0000 7fc3f2a4a700 1 Processor -- start 2024-03-26T01:56:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.903+0000 7fc3f2a4a700 1 -- start start 2024-03-26T01:56:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.903+0000 7fc3f2a4a700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec1060e0 0x7fc3ec199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.903+0000 7fc3f2a4a700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ec19a3c0 con 0x7fc3ec1060e0 2024-03-26T01:56:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.903+0000 7fc3ebfff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec1060e0 0x7fc3ec199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.903+0000 7fc3ebfff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec1060e0 0x7fc3ec199e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53862/0 (socket says 172.21.15.72:53862) 2024-03-26T01:56:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.903+0000 7fc3ebfff700 1 -- 172.21.15.72:0/1249844685 learned_addr learned my addr 172.21.15.72:0/1249844685 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.903+0000 7fc3ebfff700 1 -- 172.21.15.72:0/1249844685 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3dc009550 con 0x7fc3ec1060e0 2024-03-26T01:56:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.904+0000 7fc3ebfff700 1 --2- 172.21.15.72:0/1249844685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec1060e0 0x7fc3ec199e80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc000c00 tx=0x7fc3dc011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:18.761 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.904+0000 7fc3e9ffb700 1 -- 172.21.15.72:0/1249844685 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc3dc0036a0 con 0x7fc3ec1060e0 2024-03-26T01:56:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.904+0000 7fc3e9ffb700 1 -- 172.21.15.72:0/1249844685 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fc3dc024440 con 0x7fc3ec1060e0 2024-03-26T01:56:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.904+0000 7fc3e9ffb700 1 -- 172.21.15.72:0/1249844685 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc3dc01a5c0 con 0x7fc3ec1060e0 2024-03-26T01:56:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.904+0000 7fc3f2a4a700 1 -- 172.21.15.72:0/1249844685 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3ec19a5c0 con 0x7fc3ec1060e0 2024-03-26T01:56:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.904+0000 7fc3f2a4a700 1 -- 172.21.15.72:0/1249844685 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3ec19aa60 con 0x7fc3ec1060e0 2024-03-26T01:56:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.905+0000 7fc3e9ffb700 1 -- 172.21.15.72:0/1249844685 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) v1 ==== 45893+0+0 (secure 0 0 0) 0x7fc3dc003800 con 0x7fc3ec1060e0 2024-03-26T01:56:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.905+0000 7fc3f2a4a700 1 -- 172.21.15.72:0/1249844685 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3ec10a510 con 0x7fc3ec1060e0 2024-03-26T01:56:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.905+0000 7fc3e9ffb700 1 --2- 172.21.15.72:0/1249844685 >> v2:172.21.15.72:6800/3912119602 conn(0x7fc3d8038ef0 0x7fc3d803b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.905+0000 7fc3e9ffb700 1 -- 172.21.15.72:0/1249844685 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7fc3dc04cab0 con 0x7fc3ec1060e0 2024-03-26T01:56:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.905+0000 7fc3e3fff700 1 --2- 172.21.15.72:0/1249844685 >> v2:172.21.15.72:6800/3912119602 conn(0x7fc3d8038ef0 0x7fc3d803b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.905+0000 7fc3e3fff700 1 --2- 172.21.15.72:0/1249844685 >> v2:172.21.15.72:6800/3912119602 conn(0x7fc3d8038ef0 0x7fc3d803b3b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4006fd0 tx=0x7fc3d4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:17.910+0000 7fc3e9ffb700 1 -- 172.21.15.72:0/1249844685 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fc3dc022340 con 0x7fc3ec1060e0 2024-03-26T01:56:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:18.105+0000 7fc3f2a4a700 1 -- 172.21.15.72:0/1249844685 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fc3ec19d420 con 0x7fc3ec1060e0 2024-03-26T01:56:18.762 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:18.108+0000 7fc3e9ffb700 1 -- 172.21.15.72:0/1249844685 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fc3dc018be0 con 0x7fc3ec1060e0 2024-03-26T01:56:18.763 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:18.110+0000 7fc3f2a4a700 1 -- 172.21.15.72:0/1249844685 >> v2:172.21.15.72:6800/3912119602 conn(0x7fc3d8038ef0 msgr2=0x7fc3d803b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:18.763 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:18.110+0000 7fc3f2a4a700 1 --2- 172.21.15.72:0/1249844685 >> v2:172.21.15.72:6800/3912119602 conn(0x7fc3d8038ef0 0x7fc3d803b3b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4006fd0 tx=0x7fc3d4006e40 comp rx=0 tx=0).stop 2024-03-26T01:56:18.763 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:18.110+0000 7fc3f2a4a700 1 -- 172.21.15.72:0/1249844685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec1060e0 msgr2=0x7fc3ec199e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:18.763 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:18.110+0000 7fc3f2a4a700 1 --2- 172.21.15.72:0/1249844685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec1060e0 0x7fc3ec199e80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc000c00 tx=0x7fc3dc011eb0 comp rx=0 tx=0).stop 2024-03-26T01:56:18.763 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:18.110+0000 7fc3f2a4a700 1 -- 172.21.15.72:0/1249844685 shutdown_connections 2024-03-26T01:56:18.763 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:18.110+0000 7fc3f2a4a700 1 --2- 172.21.15.72:0/1249844685 >> v2:172.21.15.72:6800/3912119602 conn(0x7fc3d8038ef0 0x7fc3d803b3b0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:18.763 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:18.110+0000 7fc3f2a4a700 1 --2- 172.21.15.72:0/1249844685 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc3ec1060e0 0x7fc3ec199e80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:18.763 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:18.110+0000 7fc3f2a4a700 1 -- 172.21.15.72:0/1249844685 >> 172.21.15.72:0/1249844685 conn(0x7fc3ec1016a0 msgr2=0x7fc3ec102190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:18.763 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:18.110+0000 7fc3f2a4a700 1 -- 172.21.15.72:0/1249844685 shutdown_connections 2024-03-26T01:56:18.763 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:18.110+0000 7fc3f2a4a700 1 -- 172.21.15.72:0/1249844685 wait complete. 2024-03-26T01:56:18.763 INFO:teuthology.orchestra.run.smithi072.stdout:Enabling the dashboard module... 2024-03-26T01:56:19.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1249844685' entity='client.admin' 2024-03-26T01:56:19.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:19 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' 2024-03-26T01:56:20.440 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3837171258' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-26T01:56:20.440 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:20 smithi072 ceph-mon[30177]: from='mgr.14120 172.21.15.72:0/1097248136' entity='mgr.y' 2024-03-26T01:56:20.440 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:20 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ignoring --setuser ceph since I am not root 2024-03-26T01:56:20.440 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:20 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ignoring --setgroup ceph since I am not root 2024-03-26T01:56:20.733 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.220+0000 7fb57e0c0700 1 Processor -- start 2024-03-26T01:56:20.733 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.220+0000 7fb57e0c0700 1 -- start start 2024-03-26T01:56:20.733 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.220+0000 7fb57e0c0700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5781049c0 0x7fb578104de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:20.733 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.220+0000 7fb57e0c0700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb578079680 con 0x7fb5781049c0 2024-03-26T01:56:20.733 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.220+0000 7fb5777fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5781049c0 0x7fb578104de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:20.734 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.220+0000 7fb5777fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5781049c0 0x7fb578104de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53872/0 (socket says 172.21.15.72:53872) 2024-03-26T01:56:20.734 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.220+0000 7fb5777fe700 1 -- 172.21.15.72:0/3166230537 learned_addr learned my addr 172.21.15.72:0/3166230537 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:20.734 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.221+0000 7fb5777fe700 1 -- 172.21.15.72:0/3166230537 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5780797c0 con 0x7fb5781049c0 2024-03-26T01:56:20.734 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.221+0000 7fb5777fe700 1 --2- 172.21.15.72:0/3166230537 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5781049c0 0x7fb578104de0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb5680098a0 tx=0x7fb568009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0e9cc8c87446ce5 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:20.734 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.221+0000 7fb5767fc700 1 -- 172.21.15.72:0/3166230537 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb568004160 con 0x7fb5781049c0 2024-03-26T01:56:20.734 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.221+0000 7fb5767fc700 1 -- 172.21.15.72:0/3166230537 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fb5680042c0 con 0x7fb5781049c0 2024-03-26T01:56:20.734 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.221+0000 7fb5767fc700 1 -- 172.21.15.72:0/3166230537 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb568011960 con 0x7fb5781049c0 2024-03-26T01:56:20.734 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.221+0000 7fb57e0c0700 1 -- 172.21.15.72:0/3166230537 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5781049c0 msgr2=0x7fb578104de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:20.734 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.221+0000 7fb57e0c0700 1 --2- 172.21.15.72:0/3166230537 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5781049c0 0x7fb578104de0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb5680098a0 tx=0x7fb568009bb0 comp rx=0 tx=0).stop 2024-03-26T01:56:20.734 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.222+0000 7fb57e0c0700 1 -- 172.21.15.72:0/3166230537 shutdown_connections 2024-03-26T01:56:20.734 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.222+0000 7fb57e0c0700 1 --2- 172.21.15.72:0/3166230537 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5781049c0 0x7fb578104de0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:20.734 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.222+0000 7fb57e0c0700 1 -- 172.21.15.72:0/3166230537 >> 172.21.15.72:0/3166230537 conn(0x7fb5781005a0 msgr2=0x7fb5781029c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:20.734 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.222+0000 7fb57e0c0700 1 -- 172.21.15.72:0/3166230537 shutdown_connections 2024-03-26T01:56:20.735 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.222+0000 7fb57e0c0700 1 -- 172.21.15.72:0/3166230537 wait complete. 2024-03-26T01:56:20.735 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.222+0000 7fb57e0c0700 1 Processor -- start 2024-03-26T01:56:20.735 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.223+0000 7fb57e0c0700 1 -- start start 2024-03-26T01:56:20.735 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.223+0000 7fb57e0c0700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5781049c0 0x7fb57819c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:20.735 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.223+0000 7fb57e0c0700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57819c590 con 0x7fb5781049c0 2024-03-26T01:56:20.735 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.223+0000 7fb5777fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5781049c0 0x7fb57819c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:20.735 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.223+0000 7fb5777fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5781049c0 0x7fb57819c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53886/0 (socket says 172.21.15.72:53886) 2024-03-26T01:56:20.735 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.223+0000 7fb5777fe700 1 -- 172.21.15.72:0/3837171258 learned_addr learned my addr 172.21.15.72:0/3837171258 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:20.735 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.223+0000 7fb5777fe700 1 -- 172.21.15.72:0/3837171258 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb568009550 con 0x7fb5781049c0 2024-03-26T01:56:20.735 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.223+0000 7fb5777fe700 1 --2- 172.21.15.72:0/3837171258 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5781049c0 0x7fb57819c050 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb568006af0 tx=0x7fb5680119e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:20.735 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.223+0000 7fb574ff9700 1 -- 172.21.15.72:0/3837171258 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb5680039b0 con 0x7fb5781049c0 2024-03-26T01:56:20.735 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.223+0000 7fb574ff9700 1 -- 172.21.15.72:0/3837171258 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fb568003b10 con 0x7fb5781049c0 2024-03-26T01:56:20.735 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.223+0000 7fb574ff9700 1 -- 172.21.15.72:0/3837171258 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb568019680 con 0x7fb5781049c0 2024-03-26T01:56:20.735 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.223+0000 7fb57e0c0700 1 -- 172.21.15.72:0/3837171258 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb57819c790 con 0x7fb5781049c0 2024-03-26T01:56:20.736 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.224+0000 7fb57e0c0700 1 -- 172.21.15.72:0/3837171258 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb57819cc30 con 0x7fb5781049c0 2024-03-26T01:56:20.736 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.225+0000 7fb57e0c0700 1 -- 172.21.15.72:0/3837171258 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb578195cf0 con 0x7fb5781049c0 2024-03-26T01:56:20.736 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.225+0000 7fb574ff9700 1 -- 172.21.15.72:0/3837171258 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 8) v1 ==== 45893+0+0 (secure 0 0 0) 0x7fb568019800 con 0x7fb5781049c0 2024-03-26T01:56:20.736 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.225+0000 7fb574ff9700 1 --2- 172.21.15.72:0/3837171258 >> v2:172.21.15.72:6800/3912119602 conn(0x7fb560038ea0 0x7fb56003b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:20.736 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.225+0000 7fb574ff9700 1 -- 172.21.15.72:0/3837171258 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7fb56804d220 con 0x7fb5781049c0 2024-03-26T01:56:20.736 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.225+0000 7fb576ffd700 1 --2- 172.21.15.72:0/3837171258 >> v2:172.21.15.72:6800/3912119602 conn(0x7fb560038ea0 0x7fb56003b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:20.736 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.225+0000 7fb576ffd700 1 --2- 172.21.15.72:0/3837171258 >> v2:172.21.15.72:6800/3912119602 conn(0x7fb560038ea0 0x7fb56003b360 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb56c006fd0 tx=0x7fb56c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:20.736 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.230+0000 7fb574ff9700 1 -- 172.21.15.72:0/3837171258 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb568023390 con 0x7fb5781049c0 2024-03-26T01:56:20.736 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:19.497+0000 7fb57e0c0700 1 -- 172.21.15.72:0/3837171258 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fb578106040 con 0x7fb5781049c0 2024-03-26T01:56:20.736 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:20.113+0000 7fb574ff9700 1 -- 172.21.15.72:0/3837171258 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fb56800b3e0 con 0x7fb5781049c0 2024-03-26T01:56:20.736 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:20.115+0000 7fb574ff9700 1 -- 172.21.15.72:0/3837171258 <== mon.0 v2:172.21.15.72:3300/0 8 ==== mgrmap(e 9) v1 ==== 45906+0+0 (secure 0 0 0) 0x7fb56801a430 con 0x7fb5781049c0 2024-03-26T01:56:20.736 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:20.116+0000 7fb57e0c0700 1 -- 172.21.15.72:0/3837171258 >> v2:172.21.15.72:6800/3912119602 conn(0x7fb560038ea0 msgr2=0x7fb56003b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:20.736 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:20.116+0000 7fb57e0c0700 1 --2- 172.21.15.72:0/3837171258 >> v2:172.21.15.72:6800/3912119602 conn(0x7fb560038ea0 0x7fb56003b360 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb56c006fd0 tx=0x7fb56c006e40 comp rx=0 tx=0).stop 2024-03-26T01:56:20.736 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:20.116+0000 7fb57e0c0700 1 -- 172.21.15.72:0/3837171258 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5781049c0 msgr2=0x7fb57819c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:20.737 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:20.116+0000 7fb57e0c0700 1 --2- 172.21.15.72:0/3837171258 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5781049c0 0x7fb57819c050 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb568006af0 tx=0x7fb5680119e0 comp rx=0 tx=0).stop 2024-03-26T01:56:20.737 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:20.116+0000 7fb57e0c0700 1 -- 172.21.15.72:0/3837171258 shutdown_connections 2024-03-26T01:56:20.737 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:20.116+0000 7fb57e0c0700 1 --2- 172.21.15.72:0/3837171258 >> v2:172.21.15.72:6800/3912119602 conn(0x7fb560038ea0 0x7fb56003b360 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:20.737 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:20.116+0000 7fb57e0c0700 1 --2- 172.21.15.72:0/3837171258 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb5781049c0 0x7fb57819c050 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:20.737 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:20.116+0000 7fb57e0c0700 1 -- 172.21.15.72:0/3837171258 >> 172.21.15.72:0/3837171258 conn(0x7fb5781005a0 msgr2=0x7fb578192e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:20.737 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:20.116+0000 7fb57e0c0700 1 -- 172.21.15.72:0/3837171258 shutdown_connections 2024-03-26T01:56:20.737 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:20.116+0000 7fb57e0c0700 1 -- 172.21.15.72:0/3837171258 wait complete. 2024-03-26T01:56:20.968 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:20 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:20.714+0000 7fe3b7c90200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T01:56:20.968 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:20 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:20.812+0000 7fe3b7c90200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T01:56:21.400 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3837171258' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-26T01:56:21.400 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:21 smithi072 ceph-mon[30177]: mgrmap e9: y(active, since 19s) 2024-03-26T01:56:21.718 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:21 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:21.506+0000 7fe3b7c90200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T01:56:21.719 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:21 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:21.605+0000 7fe3b7c90200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T01:56:22.114 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:21 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:21.748+0000 7fe3b7c90200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T01:56:22.114 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:21 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:21.845+0000 7fe3b7c90200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T01:56:22.114 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:21 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:21.930+0000 7fe3b7c90200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T01:56:22.394 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:22 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1357130720' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T01:56:22.394 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:22 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:22.196+0000 7fe3b7c90200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T01:56:22.394 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:22 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:22.307+0000 7fe3b7c90200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T01:56:22.394 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:22 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:22.393+0000 7fe3b7c90200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T01:56:22.517 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout { 2024-03-26T01:56:22.517 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-26T01:56:22.517 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T01:56:22.517 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-26T01:56:22.517 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T01:56:22.517 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout } 2024-03-26T01:56:22.517 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.194+0000 7fa404144700 1 Processor -- start 2024-03-26T01:56:22.518 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.194+0000 7fa404144700 1 -- start start 2024-03-26T01:56:22.518 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.194+0000 7fa404144700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3fc104fd0 0x7fa3fc1053f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:22.518 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.194+0000 7fa404144700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc1059c0 con 0x7fa3fc104fd0 2024-03-26T01:56:22.518 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.194+0000 7fa401ee0700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3fc104fd0 0x7fa3fc1053f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:22.518 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.194+0000 7fa401ee0700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3fc104fd0 0x7fa3fc1053f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53920/0 (socket says 172.21.15.72:53920) 2024-03-26T01:56:22.518 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.194+0000 7fa401ee0700 1 -- 172.21.15.72:0/895636328 learned_addr learned my addr 172.21.15.72:0/895636328 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:22.518 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.195+0000 7fa401ee0700 1 -- 172.21.15.72:0/895636328 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3fc1061d0 con 0x7fa3fc104fd0 2024-03-26T01:56:22.518 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.195+0000 7fa401ee0700 1 --2- 172.21.15.72:0/895636328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3fc104fd0 0x7fa3fc1053f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa3f80098a0 tx=0x7fa3f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b2ed271513a4d70 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:22.518 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.195+0000 7fa400ede700 1 -- 172.21.15.72:0/895636328 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa3f8004160 con 0x7fa3fc104fd0 2024-03-26T01:56:22.518 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.195+0000 7fa400ede700 1 -- 172.21.15.72:0/895636328 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fa3f80042c0 con 0x7fa3fc104fd0 2024-03-26T01:56:22.518 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.195+0000 7fa400ede700 1 -- 172.21.15.72:0/895636328 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa3f8011960 con 0x7fa3fc104fd0 2024-03-26T01:56:22.519 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.195+0000 7fa404144700 1 -- 172.21.15.72:0/895636328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3fc104fd0 msgr2=0x7fa3fc1053f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:22.519 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.195+0000 7fa404144700 1 --2- 172.21.15.72:0/895636328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3fc104fd0 0x7fa3fc1053f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa3f80098a0 tx=0x7fa3f8009bb0 comp rx=0 tx=0).stop 2024-03-26T01:56:22.519 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.196+0000 7fa404144700 1 -- 172.21.15.72:0/895636328 shutdown_connections 2024-03-26T01:56:22.519 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.196+0000 7fa404144700 1 --2- 172.21.15.72:0/895636328 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3fc104fd0 0x7fa3fc1053f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:22.519 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.196+0000 7fa404144700 1 -- 172.21.15.72:0/895636328 >> 172.21.15.72:0/895636328 conn(0x7fa3fc100590 msgr2=0x7fa3fc1029b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:22.519 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.196+0000 7fa404144700 1 -- 172.21.15.72:0/895636328 shutdown_connections 2024-03-26T01:56:22.519 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.196+0000 7fa404144700 1 -- 172.21.15.72:0/895636328 wait complete. 2024-03-26T01:56:22.519 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.196+0000 7fa404144700 1 Processor -- start 2024-03-26T01:56:22.519 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.196+0000 7fa404144700 1 -- start start 2024-03-26T01:56:22.519 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.196+0000 7fa404144700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3fc104fd0 0x7fa3fc199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:22.519 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.196+0000 7fa404144700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3fc19a3f0 con 0x7fa3fc104fd0 2024-03-26T01:56:22.519 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.196+0000 7fa401ee0700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3fc104fd0 0x7fa3fc199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:22.519 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.196+0000 7fa401ee0700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3fc104fd0 0x7fa3fc199eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:53936/0 (socket says 172.21.15.72:53936) 2024-03-26T01:56:22.519 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.197+0000 7fa401ee0700 1 -- 172.21.15.72:0/1357130720 learned_addr learned my addr 172.21.15.72:0/1357130720 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:22.520 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.197+0000 7fa401ee0700 1 -- 172.21.15.72:0/1357130720 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f8009550 con 0x7fa3fc104fd0 2024-03-26T01:56:22.520 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.197+0000 7fa401ee0700 1 --2- 172.21.15.72:0/1357130720 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3fc104fd0 0x7fa3fc199eb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8006af0 tx=0x7fa3f80119e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:22.520 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.197+0000 7fa3f2ffd700 1 -- 172.21.15.72:0/1357130720 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa3f80039b0 con 0x7fa3fc104fd0 2024-03-26T01:56:22.520 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.197+0000 7fa3f2ffd700 1 -- 172.21.15.72:0/1357130720 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fa3f8003b10 con 0x7fa3fc104fd0 2024-03-26T01:56:22.520 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.197+0000 7fa404144700 1 -- 172.21.15.72:0/1357130720 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3fc19a5f0 con 0x7fa3fc104fd0 2024-03-26T01:56:22.520 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.197+0000 7fa3f2ffd700 1 -- 172.21.15.72:0/1357130720 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa3f8019670 con 0x7fa3fc104fd0 2024-03-26T01:56:22.520 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.197+0000 7fa404144700 1 -- 172.21.15.72:0/1357130720 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3fc19aa90 con 0x7fa3fc104fd0 2024-03-26T01:56:22.520 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.198+0000 7fa404144700 1 -- 172.21.15.72:0/1357130720 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3fc194070 con 0x7fa3fc104fd0 2024-03-26T01:56:22.520 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.198+0000 7fa3f2ffd700 1 -- 172.21.15.72:0/1357130720 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 9) v1 ==== 45906+0+0 (secure 0 0 0) 0x7fa3f801a430 con 0x7fa3fc104fd0 2024-03-26T01:56:22.520 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.198+0000 7fa3f2ffd700 1 --2- 172.21.15.72:0/1357130720 >> v2:172.21.15.72:6800/3912119602 conn(0x7fa3e8038ef0 0x7fa3e803b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:22.520 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.199+0000 7fa3f2ffd700 1 -- 172.21.15.72:0/1357130720 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7fa3f804d390 con 0x7fa3fc104fd0 2024-03-26T01:56:22.520 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.199+0000 7fa4016df700 1 -- 172.21.15.72:0/1357130720 >> v2:172.21.15.72:6800/3912119602 conn(0x7fa3e8038ef0 msgr2=0x7fa3e803b3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/3912119602 2024-03-26T01:56:22.520 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.199+0000 7fa4016df700 1 --2- 172.21.15.72:0/1357130720 >> v2:172.21.15.72:6800/3912119602 conn(0x7fa3e8038ef0 0x7fa3e803b3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T01:56:22.520 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.203+0000 7fa3f2ffd700 1 -- 172.21.15.72:0/1357130720 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa3f801aa50 con 0x7fa3fc104fd0 2024-03-26T01:56:22.520 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.400+0000 7fa4016df700 1 -- 172.21.15.72:0/1357130720 >> v2:172.21.15.72:6800/3912119602 conn(0x7fa3e8038ef0 msgr2=0x7fa3e803b3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/3912119602 2024-03-26T01:56:22.521 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.400+0000 7fa4016df700 1 --2- 172.21.15.72:0/1357130720 >> v2:172.21.15.72:6800/3912119602 conn(0x7fa3e8038ef0 0x7fa3e803b3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T01:56:22.521 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.466+0000 7fa404144700 1 -- 172.21.15.72:0/1357130720 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fa3fc109400 con 0x7fa3fc104fd0 2024-03-26T01:56:22.521 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.466+0000 7fa3f2ffd700 1 -- 172.21.15.72:0/1357130720 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fa3f8023390 con 0x7fa3fc104fd0 2024-03-26T01:56:22.521 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.468+0000 7fa404144700 1 -- 172.21.15.72:0/1357130720 >> v2:172.21.15.72:6800/3912119602 conn(0x7fa3e8038ef0 msgr2=0x7fa3e803b3b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T01:56:22.521 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.468+0000 7fa404144700 1 --2- 172.21.15.72:0/1357130720 >> v2:172.21.15.72:6800/3912119602 conn(0x7fa3e8038ef0 0x7fa3e803b3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:22.521 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.468+0000 7fa404144700 1 -- 172.21.15.72:0/1357130720 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3fc104fd0 msgr2=0x7fa3fc199eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:22.521 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.468+0000 7fa404144700 1 --2- 172.21.15.72:0/1357130720 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3fc104fd0 0x7fa3fc199eb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8006af0 tx=0x7fa3f80119e0 comp rx=0 tx=0).stop 2024-03-26T01:56:22.521 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.468+0000 7fa404144700 1 -- 172.21.15.72:0/1357130720 shutdown_connections 2024-03-26T01:56:22.521 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.468+0000 7fa404144700 1 --2- 172.21.15.72:0/1357130720 >> v2:172.21.15.72:6800/3912119602 conn(0x7fa3e8038ef0 0x7fa3e803b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:22.521 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.468+0000 7fa404144700 1 --2- 172.21.15.72:0/1357130720 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa3fc104fd0 0x7fa3fc199eb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:22.521 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.468+0000 7fa404144700 1 -- 172.21.15.72:0/1357130720 >> 172.21.15.72:0/1357130720 conn(0x7fa3fc100590 msgr2=0x7fa3fc101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:22.521 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.469+0000 7fa404144700 1 -- 172.21.15.72:0/1357130720 shutdown_connections 2024-03-26T01:56:22.521 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:21.469+0000 7fa404144700 1 -- 172.21.15.72:0/1357130720 wait complete. 2024-03-26T01:56:22.521 INFO:teuthology.orchestra.run.smithi072.stdout:Waiting for the mgr to restart... 2024-03-26T01:56:22.522 INFO:teuthology.orchestra.run.smithi072.stdout:Waiting for mgr epoch 9... 2024-03-26T01:56:22.673 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:22 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:22.490+0000 7fe3b7c90200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T01:56:22.968 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:22 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:22.673+0000 7fe3b7c90200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T01:56:23.718 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:23 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:23.652+0000 7fe3b7c90200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T01:56:24.134 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:23 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:23.783+0000 7fe3b7c90200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T01:56:24.134 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:24.134+0000 7fe3b7c90200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T01:56:24.391 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:24.224+0000 7fe3b7c90200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T01:56:24.718 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:24.420+0000 7fe3b7c90200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T01:56:24.718 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:24.606+0000 7fe3b7c90200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T01:56:25.146 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:24.891+0000 7fe3b7c90200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T01:56:25.468 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:25 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:25.146+0000 7fe3b7c90200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T01:56:25.468 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:25 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:25.383+0000 7fe3b7c90200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T01:56:26.468 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:26 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:26.178+0000 7fe3b7c90200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T01:56:27.056 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:26 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:26.740+0000 7fe3b7c90200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T01:56:27.056 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:26 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:26.837+0000 7fe3b7c90200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T01:56:27.056 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 01:56:27 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T01:56:27.055+0000 7fe3b7c90200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T01:56:27.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:27 smithi072 ceph-mon[30177]: Active manager daemon y restarted 2024-03-26T01:56:27.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:27 smithi072 ceph-mon[30177]: Activating manager daemon y 2024-03-26T01:56:27.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:27 smithi072 ceph-mon[30177]: osdmap e3: 0 total, 0 up, 0 in 2024-03-26T01:56:27.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:27 smithi072 ceph-mon[30177]: mgrmap e10: y(active, starting, since 0.00345723s) 2024-03-26T01:56:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:27 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:56:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:27 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T01:56:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:27 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T01:56:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:27 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T01:56:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:27 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T01:56:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:27 smithi072 ceph-mon[30177]: Manager daemon y is now available 2024-03-26T01:56:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:27 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:27 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:28.306 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:28 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T01:56:28.306 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:28 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T01:56:28.306 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:28 smithi072 ceph-mon[30177]: mgrmap e11: y(active, since 1.00551s) 2024-03-26T01:56:29.005 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout { 2024-03-26T01:56:29.005 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-26T01:56:29.005 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T01:56:29.006 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout } 2024-03-26T01:56:29.006 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.983+0000 7f6169b8d700 1 Processor -- start 2024-03-26T01:56:29.006 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.983+0000 7f6169b8d700 1 -- start start 2024-03-26T01:56:29.006 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.983+0000 7f6169b8d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6164106b80 0x7f6164108f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:29.006 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.983+0000 7f6169b8d700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6164079f70 con 0x7f6164106b80 2024-03-26T01:56:29.006 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.983+0000 7f61637fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6164106b80 0x7f6164108f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:29.006 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.983+0000 7f61637fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6164106b80 0x7f6164108f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38682/0 (socket says 172.21.15.72:38682) 2024-03-26T01:56:29.006 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.983+0000 7f61637fe700 1 -- 172.21.15.72:0/3564078796 learned_addr learned my addr 172.21.15.72:0/3564078796 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:29.006 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.984+0000 7f61637fe700 1 -- 172.21.15.72:0/3564078796 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61641094b0 con 0x7f6164106b80 2024-03-26T01:56:29.006 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.984+0000 7f61637fe700 1 --2- 172.21.15.72:0/3564078796 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6164106b80 0x7f6164108f70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f61580098a0 tx=0x7f6158009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f75c6b274b8fffc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:29.006 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.984+0000 7f61627fc700 1 -- 172.21.15.72:0/3564078796 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6158004160 con 0x7f6164106b80 2024-03-26T01:56:29.006 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.984+0000 7f61627fc700 1 -- 172.21.15.72:0/3564078796 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f61580042c0 con 0x7f6164106b80 2024-03-26T01:56:29.006 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.984+0000 7f61627fc700 1 -- 172.21.15.72:0/3564078796 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6158011960 con 0x7f6164106b80 2024-03-26T01:56:29.007 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.985+0000 7f6169b8d700 1 -- 172.21.15.72:0/3564078796 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6164106b80 msgr2=0x7f6164108f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:29.007 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.985+0000 7f6169b8d700 1 --2- 172.21.15.72:0/3564078796 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6164106b80 0x7f6164108f70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f61580098a0 tx=0x7f6158009bb0 comp rx=0 tx=0).stop 2024-03-26T01:56:29.007 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.985+0000 7f6169b8d700 1 -- 172.21.15.72:0/3564078796 shutdown_connections 2024-03-26T01:56:29.007 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.985+0000 7f6169b8d700 1 --2- 172.21.15.72:0/3564078796 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6164106b80 0x7f6164108f70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:29.007 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.985+0000 7f6169b8d700 1 -- 172.21.15.72:0/3564078796 >> 172.21.15.72:0/3564078796 conn(0x7f6164100550 msgr2=0x7f61641029b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:29.007 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.985+0000 7f6169b8d700 1 -- 172.21.15.72:0/3564078796 shutdown_connections 2024-03-26T01:56:29.007 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.985+0000 7f6169b8d700 1 -- 172.21.15.72:0/3564078796 wait complete. 2024-03-26T01:56:29.007 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.986+0000 7f6169b8d700 1 Processor -- start 2024-03-26T01:56:29.007 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.986+0000 7f6169b8d700 1 -- start start 2024-03-26T01:56:29.007 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.986+0000 7f6169b8d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6164106b80 0x7f6164199e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:29.007 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.986+0000 7f6169b8d700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616419a360 con 0x7f6164106b80 2024-03-26T01:56:29.007 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.986+0000 7f61637fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6164106b80 0x7f6164199e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:29.007 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.986+0000 7f61637fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6164106b80 0x7f6164199e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38694/0 (socket says 172.21.15.72:38694) 2024-03-26T01:56:29.007 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.986+0000 7f61637fe700 1 -- 172.21.15.72:0/958710087 learned_addr learned my addr 172.21.15.72:0/958710087 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:29.008 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.986+0000 7f61637fe700 1 -- 172.21.15.72:0/958710087 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6158009550 con 0x7f6164106b80 2024-03-26T01:56:29.008 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.987+0000 7f61637fe700 1 --2- 172.21.15.72:0/958710087 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6164106b80 0x7f6164199e20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6158004290 tx=0x7f6158011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:29.008 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.987+0000 7f6160ff9700 1 -- 172.21.15.72:0/958710087 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f61580036a0 con 0x7f6164106b80 2024-03-26T01:56:29.008 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.987+0000 7f6160ff9700 1 -- 172.21.15.72:0/958710087 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f6158024440 con 0x7f6164106b80 2024-03-26T01:56:29.008 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.987+0000 7f6169b8d700 1 -- 172.21.15.72:0/958710087 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f616419a560 con 0x7f6164106b80 2024-03-26T01:56:29.008 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.987+0000 7f6160ff9700 1 -- 172.21.15.72:0/958710087 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f615801a5c0 con 0x7f6164106b80 2024-03-26T01:56:29.008 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.987+0000 7f6169b8d700 1 -- 172.21.15.72:0/958710087 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f616419aa00 con 0x7f6164106b80 2024-03-26T01:56:29.008 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.988+0000 7f6160ff9700 1 -- 172.21.15.72:0/958710087 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 9) v1 ==== 45906+0+0 (secure 0 0 0) 0x7f615801a880 con 0x7f6164106b80 2024-03-26T01:56:29.008 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.988+0000 7f6160ff9700 1 --2- 172.21.15.72:0/958710087 >> v2:172.21.15.72:6800/3912119602 conn(0x7f614c038ef0 0x7f614c03b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:29.008 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.988+0000 7f6160ff9700 1 -- 172.21.15.72:0/958710087 --> v2:172.21.15.72:6800/3912119602 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f614c03ba80 con 0x7f614c038ef0 2024-03-26T01:56:29.008 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.988+0000 7f6160ff9700 1 -- 172.21.15.72:0/958710087 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (secure 0 0 0) 0x7f615804caa0 con 0x7f6164106b80 2024-03-26T01:56:29.008 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.988+0000 7f6162ffd700 1 -- 172.21.15.72:0/958710087 >> v2:172.21.15.72:6800/3912119602 conn(0x7f614c038ef0 msgr2=0x7f614c03b3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/3912119602 2024-03-26T01:56:29.008 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:22.988+0000 7f6162ffd700 1 --2- 172.21.15.72:0/958710087 >> v2:172.21.15.72:6800/3912119602 conn(0x7f614c038ef0 0x7f614c03b3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T01:56:29.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:23.189+0000 7f6162ffd700 1 -- 172.21.15.72:0/958710087 >> v2:172.21.15.72:6800/3912119602 conn(0x7f614c038ef0 msgr2=0x7f614c03b3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/3912119602 2024-03-26T01:56:29.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:23.189+0000 7f6162ffd700 1 --2- 172.21.15.72:0/958710087 >> v2:172.21.15.72:6800/3912119602 conn(0x7f614c038ef0 0x7f614c03b3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T01:56:29.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:23.590+0000 7f6162ffd700 1 -- 172.21.15.72:0/958710087 >> v2:172.21.15.72:6800/3912119602 conn(0x7f614c038ef0 msgr2=0x7f614c03b3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/3912119602 2024-03-26T01:56:29.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:23.590+0000 7f6162ffd700 1 --2- 172.21.15.72:0/958710087 >> v2:172.21.15.72:6800/3912119602 conn(0x7f614c038ef0 0x7f614c03b3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-26T01:56:29.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:24.391+0000 7f6162ffd700 1 -- 172.21.15.72:0/958710087 >> v2:172.21.15.72:6800/3912119602 conn(0x7f614c038ef0 msgr2=0x7f614c03b3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/3912119602 2024-03-26T01:56:29.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:24.391+0000 7f6162ffd700 1 --2- 172.21.15.72:0/958710087 >> v2:172.21.15.72:6800/3912119602 conn(0x7f614c038ef0 0x7f614c03b3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-26T01:56:29.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:25.992+0000 7f6162ffd700 1 -- 172.21.15.72:0/958710087 >> v2:172.21.15.72:6800/3912119602 conn(0x7f614c038ef0 msgr2=0x7f614c03b3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/3912119602 2024-03-26T01:56:29.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:25.992+0000 7f6162ffd700 1 --2- 172.21.15.72:0/958710087 >> v2:172.21.15.72:6800/3912119602 conn(0x7f614c038ef0 0x7f614c03b3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-26T01:56:29.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:27.071+0000 7f6160ff9700 1 -- 172.21.15.72:0/958710087 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mgrmap(e 10) v1 ==== 45708+0+0 (secure 0 0 0) 0x7f615802d910 con 0x7f6164106b80 2024-03-26T01:56:29.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:27.071+0000 7f6160ff9700 1 -- 172.21.15.72:0/958710087 >> v2:172.21.15.72:6800/3912119602 conn(0x7f614c038ef0 msgr2=0x7f614c03b3b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T01:56:29.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:27.071+0000 7f6160ff9700 1 --2- 172.21.15.72:0/958710087 >> v2:172.21.15.72:6800/3912119602 conn(0x7f614c038ef0 0x7f614c03b3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:29.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:28.072+0000 7f6160ff9700 1 -- 172.21.15.72:0/958710087 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mgrmap(e 11) v1 ==== 45800+0+0 (secure 0 0 0) 0x7f615804e300 con 0x7f6164106b80 2024-03-26T01:56:29.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:28.072+0000 7f6160ff9700 1 --2- 172.21.15.72:0/958710087 >> v2:172.21.15.72:6800/2923394687 conn(0x7f614c03c900 0x7f614c03ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:29.009 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:28.072+0000 7f6160ff9700 1 -- 172.21.15.72:0/958710087 --> v2:172.21.15.72:6800/2923394687 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f614c03ba80 con 0x7f614c03c900 2024-03-26T01:56:29.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:28.073+0000 7f6162ffd700 1 --2- 172.21.15.72:0/958710087 >> v2:172.21.15.72:6800/2923394687 conn(0x7f614c03c900 0x7f614c03ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:29.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:28.073+0000 7f6162ffd700 1 --2- 172.21.15.72:0/958710087 >> v2:172.21.15.72:6800/2923394687 conn(0x7f614c03c900 0x7f614c03ecf0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6154003a10 tx=0x7f61540092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:29.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:28.074+0000 7f6160ff9700 1 -- 172.21.15.72:0/958710087 <== mgr.14152 v2:172.21.15.72:6800/2923394687 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f614c03ba80 con 0x7f614c03c900 2024-03-26T01:56:29.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:28.080+0000 7f6169b8d700 1 -- 172.21.15.72:0/958710087 --> v2:172.21.15.72:6800/2923394687 -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f61640636b0 con 0x7f614c03c900 2024-03-26T01:56:29.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:28.080+0000 7f6160ff9700 1 -- 172.21.15.72:0/958710087 <== mgr.14152 v2:172.21.15.72:6800/2923394687 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f61640636b0 con 0x7f614c03c900 2024-03-26T01:56:29.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:28.081+0000 7f6169b8d700 1 -- 172.21.15.72:0/958710087 >> v2:172.21.15.72:6800/2923394687 conn(0x7f614c03c900 msgr2=0x7f614c03ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:29.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:28.081+0000 7f6169b8d700 1 --2- 172.21.15.72:0/958710087 >> v2:172.21.15.72:6800/2923394687 conn(0x7f614c03c900 0x7f614c03ecf0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6154003a10 tx=0x7f61540092b0 comp rx=0 tx=0).stop 2024-03-26T01:56:29.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:28.081+0000 7f6169b8d700 1 -- 172.21.15.72:0/958710087 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6164106b80 msgr2=0x7f6164199e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:29.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:28.081+0000 7f6169b8d700 1 --2- 172.21.15.72:0/958710087 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6164106b80 0x7f6164199e20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6158004290 tx=0x7f6158011eb0 comp rx=0 tx=0).stop 2024-03-26T01:56:29.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:28.081+0000 7f6169b8d700 1 -- 172.21.15.72:0/958710087 shutdown_connections 2024-03-26T01:56:29.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:28.081+0000 7f6169b8d700 1 --2- 172.21.15.72:0/958710087 >> v2:172.21.15.72:6800/2923394687 conn(0x7f614c03c900 0x7f614c03ecf0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:29.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:28.081+0000 7f6169b8d700 1 --2- 172.21.15.72:0/958710087 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f6164106b80 0x7f6164199e20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:29.010 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:28.081+0000 7f6169b8d700 1 --2- 172.21.15.72:0/958710087 >> v2:172.21.15.72:6800/3912119602 conn(0x7f614c038ef0 0x7f614c03b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:29.011 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:28.081+0000 7f6169b8d700 1 -- 172.21.15.72:0/958710087 >> 172.21.15.72:0/958710087 conn(0x7f6164100550 msgr2=0x7f61641010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:29.011 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:28.081+0000 7f6169b8d700 1 -- 172.21.15.72:0/958710087 shutdown_connections 2024-03-26T01:56:29.011 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:28.081+0000 7f6169b8d700 1 -- 172.21.15.72:0/958710087 wait complete. 2024-03-26T01:56:29.011 INFO:teuthology.orchestra.run.smithi072.stdout:mgr epoch 9 is available 2024-03-26T01:56:29.011 INFO:teuthology.orchestra.run.smithi072.stdout:Generating a dashboard self-signed certificate... 2024-03-26T01:56:29.259 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:29 smithi072 ceph-mon[30177]: [26/Mar/2024:01:56:27] ENGINE Bus STARTING 2024-03-26T01:56:29.259 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:29 smithi072 ceph-mon[30177]: [26/Mar/2024:01:56:27] ENGINE Serving on http://172.21.15.72:8765 2024-03-26T01:56:29.260 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:29 smithi072 ceph-mon[30177]: [26/Mar/2024:01:56:27] ENGINE Serving on https://172.21.15.72:7150 2024-03-26T01:56:29.260 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:29 smithi072 ceph-mon[30177]: [26/Mar/2024:01:56:27] ENGINE Bus STARTED 2024-03-26T01:56:29.260 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:29 smithi072 ceph-mon[30177]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T01:56:29.260 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:29 smithi072 ceph-mon[30177]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T01:56:29.260 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:29 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:30.426 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-26T01:56:30.426 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.471+0000 7fe819db8700 1 Processor -- start 2024-03-26T01:56:30.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.471+0000 7fe819db8700 1 -- start start 2024-03-26T01:56:30.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.471+0000 7fe819db8700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe814106110 0x7fe814106530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:30.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.471+0000 7fe819db8700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe814106b00 con 0x7fe814106110 2024-03-26T01:56:30.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.471+0000 7fe8137fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe814106110 0x7fe814106530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:30.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.471+0000 7fe8137fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe814106110 0x7fe814106530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38778/0 (socket says 172.21.15.72:38778) 2024-03-26T01:56:30.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.471+0000 7fe8137fe700 1 -- 172.21.15.72:0/2167439686 learned_addr learned my addr 172.21.15.72:0/2167439686 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:30.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.472+0000 7fe8137fe700 1 -- 172.21.15.72:0/2167439686 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe814107310 con 0x7fe814106110 2024-03-26T01:56:30.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.472+0000 7fe8137fe700 1 --2- 172.21.15.72:0/2167439686 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe814106110 0x7fe814106530 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc0098a0 tx=0x7fe7fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d552d1f70b8861f server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:30.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.472+0000 7fe8127fc700 1 -- 172.21.15.72:0/2167439686 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe7fc004160 con 0x7fe814106110 2024-03-26T01:56:30.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.472+0000 7fe8127fc700 1 -- 172.21.15.72:0/2167439686 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fe7fc0042c0 con 0x7fe814106110 2024-03-26T01:56:30.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.472+0000 7fe8127fc700 1 -- 172.21.15.72:0/2167439686 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe7fc011960 con 0x7fe814106110 2024-03-26T01:56:30.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.473+0000 7fe819db8700 1 -- 172.21.15.72:0/2167439686 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe814106110 msgr2=0x7fe814106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:30.427 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.473+0000 7fe819db8700 1 --2- 172.21.15.72:0/2167439686 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe814106110 0x7fe814106530 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc0098a0 tx=0x7fe7fc009bb0 comp rx=0 tx=0).stop 2024-03-26T01:56:30.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.473+0000 7fe819db8700 1 -- 172.21.15.72:0/2167439686 shutdown_connections 2024-03-26T01:56:30.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.473+0000 7fe819db8700 1 --2- 172.21.15.72:0/2167439686 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe814106110 0x7fe814106530 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:30.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.473+0000 7fe819db8700 1 -- 172.21.15.72:0/2167439686 >> 172.21.15.72:0/2167439686 conn(0x7fe8141016f0 msgr2=0x7fe814103b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:30.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.473+0000 7fe819db8700 1 -- 172.21.15.72:0/2167439686 shutdown_connections 2024-03-26T01:56:30.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.473+0000 7fe819db8700 1 -- 172.21.15.72:0/2167439686 wait complete. 2024-03-26T01:56:30.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.473+0000 7fe819db8700 1 Processor -- start 2024-03-26T01:56:30.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.474+0000 7fe819db8700 1 -- start start 2024-03-26T01:56:30.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.474+0000 7fe819db8700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe814106110 0x7fe814199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:30.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.474+0000 7fe819db8700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81419a420 con 0x7fe814106110 2024-03-26T01:56:30.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.474+0000 7fe8137fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe814106110 0x7fe814199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:30.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.474+0000 7fe8137fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe814106110 0x7fe814199ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38788/0 (socket says 172.21.15.72:38788) 2024-03-26T01:56:30.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.474+0000 7fe8137fe700 1 -- 172.21.15.72:0/1599230471 learned_addr learned my addr 172.21.15.72:0/1599230471 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:30.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.474+0000 7fe8137fe700 1 -- 172.21.15.72:0/1599230471 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc009550 con 0x7fe814106110 2024-03-26T01:56:30.428 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.474+0000 7fe8137fe700 1 --2- 172.21.15.72:0/1599230471 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe814106110 0x7fe814199ee0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc000c00 tx=0x7fe7fc011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:30.429 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.474+0000 7fe810ff9700 1 -- 172.21.15.72:0/1599230471 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe7fc0036a0 con 0x7fe814106110 2024-03-26T01:56:30.429 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.474+0000 7fe810ff9700 1 -- 172.21.15.72:0/1599230471 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fe7fc024440 con 0x7fe814106110 2024-03-26T01:56:30.429 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.474+0000 7fe810ff9700 1 -- 172.21.15.72:0/1599230471 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe7fc01a5c0 con 0x7fe814106110 2024-03-26T01:56:30.429 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.475+0000 7fe819db8700 1 -- 172.21.15.72:0/1599230471 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe81419a620 con 0x7fe814106110 2024-03-26T01:56:30.429 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.475+0000 7fe819db8700 1 -- 172.21.15.72:0/1599230471 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe81419aac0 con 0x7fe814106110 2024-03-26T01:56:30.429 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.475+0000 7fe810ff9700 1 -- 172.21.15.72:0/1599230471 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 11) v1 ==== 45800+0+0 (secure 0 0 0) 0x7fe7fc01a830 con 0x7fe814106110 2024-03-26T01:56:30.429 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.476+0000 7fe810ff9700 1 --2- 172.21.15.72:0/1599230471 >> v2:172.21.15.72:6800/2923394687 conn(0x7fe800038e20 0x7fe80003b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:30.429 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.476+0000 7fe810ff9700 1 -- 172.21.15.72:0/1599230471 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7fe7fc04d040 con 0x7fe814106110 2024-03-26T01:56:30.429 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.476+0000 7fe819db8700 1 -- 172.21.15.72:0/1599230471 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe81410a540 con 0x7fe814106110 2024-03-26T01:56:30.429 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.476+0000 7fe812ffd700 1 --2- 172.21.15.72:0/1599230471 >> v2:172.21.15.72:6800/2923394687 conn(0x7fe800038e20 0x7fe80003b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:30.429 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.476+0000 7fe812ffd700 1 --2- 172.21.15.72:0/1599230471 >> v2:172.21.15.72:6800/2923394687 conn(0x7fe800038e20 0x7fe80003b2e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe804006fd0 tx=0x7fe804006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:30.429 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.481+0000 7fe810ff9700 1 -- 172.21.15.72:0/1599230471 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe7fc01ab30 con 0x7fe814106110 2024-03-26T01:56:30.429 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.687+0000 7fe810ff9700 1 -- 172.21.15.72:0/1599230471 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mgrmap(e 12) v1 ==== 45906+0+0 (secure 0 0 0) 0x7fe7fc02d910 con 0x7fe814106110 2024-03-26T01:56:30.429 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.690+0000 7fe819db8700 1 -- 172.21.15.72:0/1599230471 --> v2:172.21.15.72:6800/2923394687 -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fe814102980 con 0x7fe800038e20 2024-03-26T01:56:30.429 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.852+0000 7fe810ff9700 1 -- 172.21.15.72:0/1599230471 <== mgr.14152 v2:172.21.15.72:6800/2923394687 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fe814102980 con 0x7fe800038e20 2024-03-26T01:56:30.430 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.854+0000 7fe819db8700 1 -- 172.21.15.72:0/1599230471 >> v2:172.21.15.72:6800/2923394687 conn(0x7fe800038e20 msgr2=0x7fe80003b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:30.430 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.854+0000 7fe819db8700 1 --2- 172.21.15.72:0/1599230471 >> v2:172.21.15.72:6800/2923394687 conn(0x7fe800038e20 0x7fe80003b2e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe804006fd0 tx=0x7fe804006e40 comp rx=0 tx=0).stop 2024-03-26T01:56:30.430 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.854+0000 7fe819db8700 1 -- 172.21.15.72:0/1599230471 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe814106110 msgr2=0x7fe814199ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:30.430 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.854+0000 7fe819db8700 1 --2- 172.21.15.72:0/1599230471 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe814106110 0x7fe814199ee0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc000c00 tx=0x7fe7fc011eb0 comp rx=0 tx=0).stop 2024-03-26T01:56:30.430 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.854+0000 7fe819db8700 1 -- 172.21.15.72:0/1599230471 shutdown_connections 2024-03-26T01:56:30.430 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.854+0000 7fe819db8700 1 --2- 172.21.15.72:0/1599230471 >> v2:172.21.15.72:6800/2923394687 conn(0x7fe800038e20 0x7fe80003b2e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:30.430 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.854+0000 7fe819db8700 1 --2- 172.21.15.72:0/1599230471 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe814106110 0x7fe814199ee0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:30.430 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.854+0000 7fe819db8700 1 -- 172.21.15.72:0/1599230471 >> 172.21.15.72:0/1599230471 conn(0x7fe8141016f0 msgr2=0x7fe814102270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:30.430 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.855+0000 7fe819db8700 1 -- 172.21.15.72:0/1599230471 shutdown_connections 2024-03-26T01:56:30.430 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:29.855+0000 7fe819db8700 1 -- 172.21.15.72:0/1599230471 wait complete. 2024-03-26T01:56:30.430 INFO:teuthology.orchestra.run.smithi072.stdout:Creating initial admin user... 2024-03-26T01:56:30.936 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:30 smithi072 ceph-mon[30177]: mgrmap e12: y(active, since 2s) 2024-03-26T01:56:30.936 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:30 smithi072 ceph-mon[30177]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:30.936 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:30 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:30.936 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:30 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:30.937 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:30 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:31.848 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:31 smithi072 ceph-mon[30177]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:31.848 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:31 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:32.127 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$CzuAXuTBBeJ0pRRWpheSUucy2jtG05bdSaK7qoAiglAXcc846inge", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711418191, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-26T01:56:32.127 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.932+0000 7fa85fac0700 1 Processor -- start 2024-03-26T01:56:32.127 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.932+0000 7fa85fac0700 1 -- start start 2024-03-26T01:56:32.127 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.932+0000 7fa85fac0700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa858106b60 0x7fa858108f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:32.127 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.932+0000 7fa85fac0700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa858079700 con 0x7fa858106b60 2024-03-26T01:56:32.128 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.933+0000 7fa85d85c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa858106b60 0x7fa858108f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:32.128 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.933+0000 7fa85d85c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa858106b60 0x7fa858108f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38790/0 (socket says 172.21.15.72:38790) 2024-03-26T01:56:32.128 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.933+0000 7fa85d85c700 1 -- 172.21.15.72:0/2417369669 learned_addr learned my addr 172.21.15.72:0/2417369669 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:32.128 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.933+0000 7fa85d85c700 1 -- 172.21.15.72:0/2417369669 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa858109490 con 0x7fa858106b60 2024-03-26T01:56:32.128 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.933+0000 7fa85d85c700 1 --2- 172.21.15.72:0/2417369669 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa858106b60 0x7fa858108f50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa8540098a0 tx=0x7fa854009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbfca733609b2187 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:32.128 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.933+0000 7fa85c85a700 1 -- 172.21.15.72:0/2417369669 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa854004160 con 0x7fa858106b60 2024-03-26T01:56:32.128 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.933+0000 7fa85c85a700 1 -- 172.21.15.72:0/2417369669 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fa8540042c0 con 0x7fa858106b60 2024-03-26T01:56:32.128 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.933+0000 7fa85c85a700 1 -- 172.21.15.72:0/2417369669 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa854011960 con 0x7fa858106b60 2024-03-26T01:56:32.128 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.934+0000 7fa85fac0700 1 -- 172.21.15.72:0/2417369669 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa858106b60 msgr2=0x7fa858108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:32.128 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.934+0000 7fa85fac0700 1 --2- 172.21.15.72:0/2417369669 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa858106b60 0x7fa858108f50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa8540098a0 tx=0x7fa854009bb0 comp rx=0 tx=0).stop 2024-03-26T01:56:32.128 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.934+0000 7fa85fac0700 1 -- 172.21.15.72:0/2417369669 shutdown_connections 2024-03-26T01:56:32.128 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.934+0000 7fa85fac0700 1 --2- 172.21.15.72:0/2417369669 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa858106b60 0x7fa858108f50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:32.128 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.934+0000 7fa85fac0700 1 -- 172.21.15.72:0/2417369669 >> 172.21.15.72:0/2417369669 conn(0x7fa858100590 msgr2=0x7fa8581029b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:32.128 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.934+0000 7fa85fac0700 1 -- 172.21.15.72:0/2417369669 shutdown_connections 2024-03-26T01:56:32.129 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.934+0000 7fa85fac0700 1 -- 172.21.15.72:0/2417369669 wait complete. 2024-03-26T01:56:32.129 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.935+0000 7fa85fac0700 1 Processor -- start 2024-03-26T01:56:32.129 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.935+0000 7fa85fac0700 1 -- start start 2024-03-26T01:56:32.129 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.935+0000 7fa85fac0700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa858106b60 0x7fa85819c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:32.129 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.935+0000 7fa85fac0700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa85819c590 con 0x7fa858106b60 2024-03-26T01:56:32.129 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.935+0000 7fa85d85c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa858106b60 0x7fa85819c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:32.129 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.935+0000 7fa85d85c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa858106b60 0x7fa85819c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:38792/0 (socket says 172.21.15.72:38792) 2024-03-26T01:56:32.129 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.935+0000 7fa85d85c700 1 -- 172.21.15.72:0/2213564554 learned_addr learned my addr 172.21.15.72:0/2213564554 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:32.129 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.935+0000 7fa85d85c700 1 -- 172.21.15.72:0/2213564554 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa854009550 con 0x7fa858106b60 2024-03-26T01:56:32.129 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.936+0000 7fa85d85c700 1 --2- 172.21.15.72:0/2213564554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa858106b60 0x7fa85819c050 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa854004290 tx=0x7fa854011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:32.129 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.936+0000 7fa84effd700 1 -- 172.21.15.72:0/2213564554 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa8540036a0 con 0x7fa858106b60 2024-03-26T01:56:32.129 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.936+0000 7fa85fac0700 1 -- 172.21.15.72:0/2213564554 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa85819c790 con 0x7fa858106b60 2024-03-26T01:56:32.129 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.936+0000 7fa84effd700 1 -- 172.21.15.72:0/2213564554 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fa854024440 con 0x7fa858106b60 2024-03-26T01:56:32.129 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.936+0000 7fa84effd700 1 -- 172.21.15.72:0/2213564554 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa85401a5c0 con 0x7fa858106b60 2024-03-26T01:56:32.130 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.936+0000 7fa85fac0700 1 -- 172.21.15.72:0/2213564554 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa85819cc30 con 0x7fa858106b60 2024-03-26T01:56:32.130 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.937+0000 7fa85fac0700 1 -- 172.21.15.72:0/2213564554 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa85810a280 con 0x7fa858106b60 2024-03-26T01:56:32.130 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.937+0000 7fa84effd700 1 -- 172.21.15.72:0/2213564554 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 12) v1 ==== 45906+0+0 (secure 0 0 0) 0x7fa85401a880 con 0x7fa858106b60 2024-03-26T01:56:32.130 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.937+0000 7fa84effd700 1 --2- 172.21.15.72:0/2213564554 >> v2:172.21.15.72:6800/2923394687 conn(0x7fa844038ef0 0x7fa84403b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:32.130 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.937+0000 7fa84effd700 1 -- 172.21.15.72:0/2213564554 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7fa8540558e0 con 0x7fa858106b60 2024-03-26T01:56:32.130 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.937+0000 7fa85d05b700 1 --2- 172.21.15.72:0/2213564554 >> v2:172.21.15.72:6800/2923394687 conn(0x7fa844038ef0 0x7fa84403b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:32.130 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.938+0000 7fa85d05b700 1 --2- 172.21.15.72:0/2213564554 >> v2:172.21.15.72:6800/2923394687 conn(0x7fa844038ef0 0x7fa84403b3b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa848006fd0 tx=0x7fa848006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:32.130 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:30.943+0000 7fa84effd700 1 -- 172.21.15.72:0/2213564554 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa854050030 con 0x7fa858106b60 2024-03-26T01:56:32.130 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:31.160+0000 7fa85fac0700 1 -- 172.21.15.72:0/2213564554 --> v2:172.21.15.72:6800/2923394687 -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fa8580634d0 con 0x7fa844038ef0 2024-03-26T01:56:32.130 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:31.384+0000 7fa84effd700 1 -- 172.21.15.72:0/2213564554 <== mgr.14152 v2:172.21.15.72:6800/2923394687 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fa8580634d0 con 0x7fa844038ef0 2024-03-26T01:56:32.130 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:31.386+0000 7fa85fac0700 1 -- 172.21.15.72:0/2213564554 >> v2:172.21.15.72:6800/2923394687 conn(0x7fa844038ef0 msgr2=0x7fa84403b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:32.130 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:31.386+0000 7fa85fac0700 1 --2- 172.21.15.72:0/2213564554 >> v2:172.21.15.72:6800/2923394687 conn(0x7fa844038ef0 0x7fa84403b3b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa848006fd0 tx=0x7fa848006e40 comp rx=0 tx=0).stop 2024-03-26T01:56:32.130 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:31.386+0000 7fa85fac0700 1 -- 172.21.15.72:0/2213564554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa858106b60 msgr2=0x7fa85819c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:32.130 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:31.386+0000 7fa85fac0700 1 --2- 172.21.15.72:0/2213564554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa858106b60 0x7fa85819c050 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa854004290 tx=0x7fa854011eb0 comp rx=0 tx=0).stop 2024-03-26T01:56:32.131 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:31.386+0000 7fa85fac0700 1 -- 172.21.15.72:0/2213564554 shutdown_connections 2024-03-26T01:56:32.131 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:31.386+0000 7fa85fac0700 1 --2- 172.21.15.72:0/2213564554 >> v2:172.21.15.72:6800/2923394687 conn(0x7fa844038ef0 0x7fa84403b3b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:32.131 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:31.386+0000 7fa85fac0700 1 --2- 172.21.15.72:0/2213564554 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa858106b60 0x7fa85819c050 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:32.131 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:31.386+0000 7fa85fac0700 1 -- 172.21.15.72:0/2213564554 >> 172.21.15.72:0/2213564554 conn(0x7fa858100590 msgr2=0x7fa858101020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:32.131 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:31.386+0000 7fa85fac0700 1 -- 172.21.15.72:0/2213564554 shutdown_connections 2024-03-26T01:56:32.131 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:31.386+0000 7fa85fac0700 1 -- 172.21.15.72:0/2213564554 wait complete. 2024-03-26T01:56:32.131 INFO:teuthology.orchestra.run.smithi072.stdout:Fetching dashboard port number... 2024-03-26T01:56:33.451 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:33 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3767944164' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-26T01:56:33.451 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:33 smithi072 ceph-mon[30177]: mgrmap e13: y(active, since 6s) 2024-03-26T01:56:33.712 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stdout 8443 2024-03-26T01:56:33.712 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.948+0000 7fbac9ace700 1 Processor -- start 2024-03-26T01:56:33.712 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.948+0000 7fbac9ace700 1 -- start start 2024-03-26T01:56:33.712 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.949+0000 7fbac9ace700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbac4104ff0 0x7fbac4105410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:33.712 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.949+0000 7fbac9ace700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac41059e0 con 0x7fbac4104ff0 2024-03-26T01:56:33.712 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.949+0000 7fbac37fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbac4104ff0 0x7fbac4105410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:33.713 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.949+0000 7fbac37fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbac4104ff0 0x7fbac4105410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39786/0 (socket says 172.21.15.72:39786) 2024-03-26T01:56:33.713 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.949+0000 7fbac37fe700 1 -- 172.21.15.72:0/2111266598 learned_addr learned my addr 172.21.15.72:0/2111266598 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:33.713 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.949+0000 7fbac37fe700 1 -- 172.21.15.72:0/2111266598 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac41061f0 con 0x7fbac4104ff0 2024-03-26T01:56:33.713 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.949+0000 7fbac37fe700 1 --2- 172.21.15.72:0/2111266598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbac4104ff0 0x7fbac4105410 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbab40098a0 tx=0x7fbab4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35af6db849dbdb33 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:33.713 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.949+0000 7fbac2ffd700 1 -- 172.21.15.72:0/2111266598 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbab4004160 con 0x7fbac4104ff0 2024-03-26T01:56:33.713 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.949+0000 7fbac2ffd700 1 -- 172.21.15.72:0/2111266598 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fbab40042c0 con 0x7fbac4104ff0 2024-03-26T01:56:33.713 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.949+0000 7fbac2ffd700 1 -- 172.21.15.72:0/2111266598 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbab4011960 con 0x7fbac4104ff0 2024-03-26T01:56:33.713 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.950+0000 7fbac9ace700 1 -- 172.21.15.72:0/2111266598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbac4104ff0 msgr2=0x7fbac4105410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:33.713 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.950+0000 7fbac9ace700 1 --2- 172.21.15.72:0/2111266598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbac4104ff0 0x7fbac4105410 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fbab40098a0 tx=0x7fbab4009bb0 comp rx=0 tx=0).stop 2024-03-26T01:56:33.713 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.950+0000 7fbac9ace700 1 -- 172.21.15.72:0/2111266598 shutdown_connections 2024-03-26T01:56:33.713 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.950+0000 7fbac9ace700 1 --2- 172.21.15.72:0/2111266598 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbac4104ff0 0x7fbac4105410 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:33.713 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.950+0000 7fbac9ace700 1 -- 172.21.15.72:0/2111266598 >> 172.21.15.72:0/2111266598 conn(0x7fbac4100570 msgr2=0x7fbac41029d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:33.713 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.950+0000 7fbac9ace700 1 -- 172.21.15.72:0/2111266598 shutdown_connections 2024-03-26T01:56:33.713 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.950+0000 7fbac9ace700 1 -- 172.21.15.72:0/2111266598 wait complete. 2024-03-26T01:56:33.714 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.951+0000 7fbac9ace700 1 Processor -- start 2024-03-26T01:56:33.714 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.951+0000 7fbac9ace700 1 -- start start 2024-03-26T01:56:33.714 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.951+0000 7fbac9ace700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbac4104ff0 0x7fbac4198d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:33.714 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.951+0000 7fbac9ace700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac41992a0 con 0x7fbac4104ff0 2024-03-26T01:56:33.714 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.951+0000 7fbac37fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbac4104ff0 0x7fbac4198d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:33.714 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.951+0000 7fbac37fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbac4104ff0 0x7fbac4198d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39802/0 (socket says 172.21.15.72:39802) 2024-03-26T01:56:33.714 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.951+0000 7fbac37fe700 1 -- 172.21.15.72:0/3767944164 learned_addr learned my addr 172.21.15.72:0/3767944164 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:33.714 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.951+0000 7fbac37fe700 1 -- 172.21.15.72:0/3767944164 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab4009550 con 0x7fbac4104ff0 2024-03-26T01:56:33.714 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.951+0000 7fbac37fe700 1 --2- 172.21.15.72:0/3767944164 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbac4104ff0 0x7fbac4198d60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbab4000c00 tx=0x7fbab4011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:33.714 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.951+0000 7fbac17fa700 1 -- 172.21.15.72:0/3767944164 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbab40036a0 con 0x7fbac4104ff0 2024-03-26T01:56:33.714 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.951+0000 7fbac9ace700 1 -- 172.21.15.72:0/3767944164 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac41994a0 con 0x7fbac4104ff0 2024-03-26T01:56:33.714 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.951+0000 7fbac17fa700 1 -- 172.21.15.72:0/3767944164 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fbab4024440 con 0x7fbac4104ff0 2024-03-26T01:56:33.714 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.951+0000 7fbac17fa700 1 -- 172.21.15.72:0/3767944164 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbab401a5c0 con 0x7fbac4104ff0 2024-03-26T01:56:33.714 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.951+0000 7fbac9ace700 1 -- 172.21.15.72:0/3767944164 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac4199940 con 0x7fbac4104ff0 2024-03-26T01:56:33.714 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.952+0000 7fbac17fa700 1 -- 172.21.15.72:0/3767944164 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 12) v1 ==== 45906+0+0 (secure 0 0 0) 0x7fbab401a880 con 0x7fbac4104ff0 2024-03-26T01:56:33.715 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.952+0000 7fbac17fa700 1 --2- 172.21.15.72:0/3767944164 >> v2:172.21.15.72:6800/2923394687 conn(0x7fbab0038ef0 0x7fbab003b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:33.715 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.952+0000 7fbac9ace700 1 -- 172.21.15.72:0/3767944164 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac4109420 con 0x7fbac4104ff0 2024-03-26T01:56:33.715 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.953+0000 7fbac17fa700 1 -- 172.21.15.72:0/3767944164 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7fbab404cf60 con 0x7fbac4104ff0 2024-03-26T01:56:33.715 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.953+0000 7fbabbfff700 1 --2- 172.21.15.72:0/3767944164 >> v2:172.21.15.72:6800/2923394687 conn(0x7fbab0038ef0 0x7fbab003b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:33.715 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.953+0000 7fbabbfff700 1 --2- 172.21.15.72:0/3767944164 >> v2:172.21.15.72:6800/2923394687 conn(0x7fbab0038ef0 0x7fbab003b3b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbaac006fd0 tx=0x7fbaac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:33.715 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:32.957+0000 7fbac17fa700 1 -- 172.21.15.72:0/3767944164 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fbab4022340 con 0x7fbac4104ff0 2024-03-26T01:56:33.715 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:33.151+0000 7fbac9ace700 1 -- 172.21.15.72:0/3767944164 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fbac419c250 con 0x7fbac4104ff0 2024-03-26T01:56:33.715 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:33.152+0000 7fbac17fa700 1 -- 172.21.15.72:0/3767944164 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fbab4018be0 con 0x7fbac4104ff0 2024-03-26T01:56:33.715 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:33.154+0000 7fbac9ace700 1 -- 172.21.15.72:0/3767944164 >> v2:172.21.15.72:6800/2923394687 conn(0x7fbab0038ef0 msgr2=0x7fbab003b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:33.715 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:33.154+0000 7fbac9ace700 1 --2- 172.21.15.72:0/3767944164 >> v2:172.21.15.72:6800/2923394687 conn(0x7fbab0038ef0 0x7fbab003b3b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbaac006fd0 tx=0x7fbaac006e40 comp rx=0 tx=0).stop 2024-03-26T01:56:33.715 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:33.154+0000 7fbac9ace700 1 -- 172.21.15.72:0/3767944164 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbac4104ff0 msgr2=0x7fbac4198d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:33.715 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:33.154+0000 7fbac9ace700 1 --2- 172.21.15.72:0/3767944164 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbac4104ff0 0x7fbac4198d60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fbab4000c00 tx=0x7fbab4011eb0 comp rx=0 tx=0).stop 2024-03-26T01:56:33.715 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:33.154+0000 7fbac9ace700 1 -- 172.21.15.72:0/3767944164 shutdown_connections 2024-03-26T01:56:33.715 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:33.154+0000 7fbac9ace700 1 --2- 172.21.15.72:0/3767944164 >> v2:172.21.15.72:6800/2923394687 conn(0x7fbab0038ef0 0x7fbab003b3b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:33.715 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:33.154+0000 7fbac9ace700 1 --2- 172.21.15.72:0/3767944164 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbac4104ff0 0x7fbac4198d60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:33.716 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:33.154+0000 7fbac9ace700 1 -- 172.21.15.72:0/3767944164 >> 172.21.15.72:0/3767944164 conn(0x7fbac4100570 msgr2=0x7fbac41010f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:33.716 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:33.154+0000 7fbac9ace700 1 -- 172.21.15.72:0/3767944164 shutdown_connections 2024-03-26T01:56:33.716 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:33.154+0000 7fbac9ace700 1 -- 172.21.15.72:0/3767944164 wait complete. 2024-03-26T01:56:33.718 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T01:56:33.719 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout disabled 2024-03-26T01:56:33.731 INFO:teuthology.orchestra.run.smithi072.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T01:56:33.732 INFO:teuthology.orchestra.run.smithi072.stdout:systemctl: stdout inactive 2024-03-26T01:56:33.732 INFO:teuthology.orchestra.run.smithi072.stdout:firewalld.service is not enabled 2024-03-26T01:56:33.732 INFO:teuthology.orchestra.run.smithi072.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-26T01:56:33.732 INFO:teuthology.orchestra.run.smithi072.stdout:Ceph Dashboard is now available at: 2024-03-26T01:56:33.732 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:56:33.732 INFO:teuthology.orchestra.run.smithi072.stdout: URL: https://smithi072.front.sepia.ceph.com:8443/ 2024-03-26T01:56:33.732 INFO:teuthology.orchestra.run.smithi072.stdout: User: admin 2024-03-26T01:56:33.732 INFO:teuthology.orchestra.run.smithi072.stdout: Password: yc6frmnyhe 2024-03-26T01:56:33.732 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:56:33.732 INFO:teuthology.orchestra.run.smithi072.stdout:Saving cluster configuration to /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config directory 2024-03-26T01:56:33.733 INFO:teuthology.orchestra.run.smithi072.stdout:Enabling autotune for osd_memory_target 2024-03-26T01:56:34.994 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.171+0000 7f144dafe700 1 Processor -- start 2024-03-26T01:56:34.994 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.171+0000 7f144dafe700 1 -- start start 2024-03-26T01:56:34.994 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.171+0000 7f144dafe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14481060f0 0x7f1448106510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:34.995 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.171+0000 7f144dafe700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1448106ae0 con 0x7f14481060f0 2024-03-26T01:56:34.995 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.171+0000 7f14477fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14481060f0 0x7f1448106510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:34.995 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.171+0000 7f14477fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14481060f0 0x7f1448106510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39816/0 (socket says 172.21.15.72:39816) 2024-03-26T01:56:34.995 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.171+0000 7f14477fe700 1 -- 172.21.15.72:0/391766114 learned_addr learned my addr 172.21.15.72:0/391766114 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:34.995 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.172+0000 7f14477fe700 1 -- 172.21.15.72:0/391766114 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14481072f0 con 0x7f14481060f0 2024-03-26T01:56:34.996 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.172+0000 7f14477fe700 1 --2- 172.21.15.72:0/391766114 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14481060f0 0x7f1448106510 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f14380098a0 tx=0x7f1438009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c573d37358d3265 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:34.999 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.172+0000 7f1446ffd700 1 -- 172.21.15.72:0/391766114 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1438004160 con 0x7f14481060f0 2024-03-26T01:56:34.999 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.172+0000 7f1446ffd700 1 -- 172.21.15.72:0/391766114 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f14380042c0 con 0x7f14481060f0 2024-03-26T01:56:34.999 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.172+0000 7f1446ffd700 1 -- 172.21.15.72:0/391766114 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1438011960 con 0x7f14481060f0 2024-03-26T01:56:34.999 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.173+0000 7f144dafe700 1 -- 172.21.15.72:0/391766114 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14481060f0 msgr2=0x7f1448106510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:35.000 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.173+0000 7f144dafe700 1 --2- 172.21.15.72:0/391766114 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14481060f0 0x7f1448106510 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f14380098a0 tx=0x7f1438009bb0 comp rx=0 tx=0).stop 2024-03-26T01:56:35.000 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.173+0000 7f144dafe700 1 -- 172.21.15.72:0/391766114 shutdown_connections 2024-03-26T01:56:35.000 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.173+0000 7f144dafe700 1 --2- 172.21.15.72:0/391766114 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14481060f0 0x7f1448106510 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:35.000 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.173+0000 7f144dafe700 1 -- 172.21.15.72:0/391766114 >> 172.21.15.72:0/391766114 conn(0x7f14481016b0 msgr2=0x7f1448103ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:35.000 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.173+0000 7f144dafe700 1 -- 172.21.15.72:0/391766114 shutdown_connections 2024-03-26T01:56:35.000 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.173+0000 7f144dafe700 1 -- 172.21.15.72:0/391766114 wait complete. 2024-03-26T01:56:35.000 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.173+0000 7f144dafe700 1 Processor -- start 2024-03-26T01:56:35.000 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.173+0000 7f144dafe700 1 -- start start 2024-03-26T01:56:35.000 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.174+0000 7f144dafe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14481060f0 0x7f1448199e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:35.000 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.174+0000 7f144dafe700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144819a3d0 con 0x7f14481060f0 2024-03-26T01:56:35.000 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.174+0000 7f14477fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14481060f0 0x7f1448199e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:35.001 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.174+0000 7f14477fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14481060f0 0x7f1448199e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39830/0 (socket says 172.21.15.72:39830) 2024-03-26T01:56:35.001 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.174+0000 7f14477fe700 1 -- 172.21.15.72:0/607427516 learned_addr learned my addr 172.21.15.72:0/607427516 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:35.001 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.174+0000 7f14477fe700 1 -- 172.21.15.72:0/607427516 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1438009550 con 0x7f14481060f0 2024-03-26T01:56:35.001 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.174+0000 7f14477fe700 1 --2- 172.21.15.72:0/607427516 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14481060f0 0x7f1448199e90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1438006af0 tx=0x7f14380119e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:35.001 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.174+0000 7f14457fa700 1 -- 172.21.15.72:0/607427516 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14380039b0 con 0x7f14481060f0 2024-03-26T01:56:35.001 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.174+0000 7f14457fa700 1 -- 172.21.15.72:0/607427516 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f1438003b10 con 0x7f14481060f0 2024-03-26T01:56:35.001 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.174+0000 7f144dafe700 1 -- 172.21.15.72:0/607427516 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f144819a5d0 con 0x7f14481060f0 2024-03-26T01:56:35.001 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.174+0000 7f14457fa700 1 -- 172.21.15.72:0/607427516 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1438019670 con 0x7f14481060f0 2024-03-26T01:56:35.001 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.174+0000 7f144dafe700 1 -- 172.21.15.72:0/607427516 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f144819aa70 con 0x7f14481060f0 2024-03-26T01:56:35.001 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.175+0000 7f14457fa700 1 -- 172.21.15.72:0/607427516 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) v1 ==== 45949+0+0 (secure 0 0 0) 0x7f14380198e0 con 0x7f14481060f0 2024-03-26T01:56:35.001 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.175+0000 7f144dafe700 1 -- 172.21.15.72:0/607427516 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f144810a520 con 0x7f14481060f0 2024-03-26T01:56:35.001 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.175+0000 7f14457fa700 1 --2- 172.21.15.72:0/607427516 >> v2:172.21.15.72:6800/2923394687 conn(0x7f1434038f90 0x7f143403b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:35.001 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.175+0000 7f14457fa700 1 -- 172.21.15.72:0/607427516 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7f143804d8f0 con 0x7f14481060f0 2024-03-26T01:56:35.001 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.175+0000 7f143effd700 1 --2- 172.21.15.72:0/607427516 >> v2:172.21.15.72:6800/2923394687 conn(0x7f1434038f90 0x7f143403b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:35.002 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.176+0000 7f143effd700 1 --2- 172.21.15.72:0/607427516 >> v2:172.21.15.72:6800/2923394687 conn(0x7f1434038f90 0x7f143403b450 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1430006fd0 tx=0x7f1430006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:35.002 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.180+0000 7f14457fa700 1 -- 172.21.15.72:0/607427516 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f143801aa50 con 0x7f14481060f0 2024-03-26T01:56:35.002 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.371+0000 7f144dafe700 1 -- 172.21.15.72:0/607427516 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f144819d430 con 0x7f14481060f0 2024-03-26T01:56:35.002 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.371+0000 7f14457fa700 1 -- 172.21.15.72:0/607427516 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f1438023390 con 0x7f14481060f0 2024-03-26T01:56:35.002 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.373+0000 7f144dafe700 1 -- 172.21.15.72:0/607427516 >> v2:172.21.15.72:6800/2923394687 conn(0x7f1434038f90 msgr2=0x7f143403b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:35.002 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.373+0000 7f144dafe700 1 --2- 172.21.15.72:0/607427516 >> v2:172.21.15.72:6800/2923394687 conn(0x7f1434038f90 0x7f143403b450 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1430006fd0 tx=0x7f1430006e40 comp rx=0 tx=0).stop 2024-03-26T01:56:35.002 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.373+0000 7f144dafe700 1 -- 172.21.15.72:0/607427516 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14481060f0 msgr2=0x7f1448199e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:35.002 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.373+0000 7f144dafe700 1 --2- 172.21.15.72:0/607427516 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14481060f0 0x7f1448199e90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1438006af0 tx=0x7f14380119e0 comp rx=0 tx=0).stop 2024-03-26T01:56:35.002 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.373+0000 7f144dafe700 1 -- 172.21.15.72:0/607427516 shutdown_connections 2024-03-26T01:56:35.002 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.373+0000 7f144dafe700 1 --2- 172.21.15.72:0/607427516 >> v2:172.21.15.72:6800/2923394687 conn(0x7f1434038f90 0x7f143403b450 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:35.002 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.373+0000 7f144dafe700 1 --2- 172.21.15.72:0/607427516 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14481060f0 0x7f1448199e90 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:35.002 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.373+0000 7f144dafe700 1 -- 172.21.15.72:0/607427516 >> 172.21.15.72:0/607427516 conn(0x7f14481016b0 msgr2=0x7f14481021a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:35.002 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.373+0000 7f144dafe700 1 -- 172.21.15.72:0/607427516 shutdown_connections 2024-03-26T01:56:35.003 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:34.373+0000 7f144dafe700 1 -- 172.21.15.72:0/607427516 wait complete. 2024-03-26T01:56:36.368 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.468+0000 7fca2ba6b700 1 Processor -- start 2024-03-26T01:56:36.368 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.468+0000 7fca2ba6b700 1 -- start start 2024-03-26T01:56:36.368 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.468+0000 7fca2ba6b700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca24105020 0x7fca24105440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:36.368 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.468+0000 7fca2ba6b700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca24105a10 con 0x7fca24105020 2024-03-26T01:56:36.369 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.468+0000 7fca29807700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca24105020 0x7fca24105440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:36.369 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.468+0000 7fca29807700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca24105020 0x7fca24105440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39834/0 (socket says 172.21.15.72:39834) 2024-03-26T01:56:36.369 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.468+0000 7fca29807700 1 -- 172.21.15.72:0/3545243845 learned_addr learned my addr 172.21.15.72:0/3545243845 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:36.369 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.469+0000 7fca29807700 1 -- 172.21.15.72:0/3545243845 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca24106220 con 0x7fca24105020 2024-03-26T01:56:36.369 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.469+0000 7fca29807700 1 --2- 172.21.15.72:0/3545243845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca24105020 0x7fca24105440 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fca200098a0 tx=0x7fca20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b8235d2d5304eed server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:36.369 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.469+0000 7fca28805700 1 -- 172.21.15.72:0/3545243845 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca20004160 con 0x7fca24105020 2024-03-26T01:56:36.369 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.469+0000 7fca28805700 1 -- 172.21.15.72:0/3545243845 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fca200042c0 con 0x7fca24105020 2024-03-26T01:56:36.369 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.469+0000 7fca28805700 1 -- 172.21.15.72:0/3545243845 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca20011960 con 0x7fca24105020 2024-03-26T01:56:36.369 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.469+0000 7fca2ba6b700 1 -- 172.21.15.72:0/3545243845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca24105020 msgr2=0x7fca24105440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:36.369 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.469+0000 7fca2ba6b700 1 --2- 172.21.15.72:0/3545243845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca24105020 0x7fca24105440 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fca200098a0 tx=0x7fca20009bb0 comp rx=0 tx=0).stop 2024-03-26T01:56:36.369 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.470+0000 7fca2ba6b700 1 -- 172.21.15.72:0/3545243845 shutdown_connections 2024-03-26T01:56:36.369 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.470+0000 7fca2ba6b700 1 --2- 172.21.15.72:0/3545243845 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca24105020 0x7fca24105440 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:36.370 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.470+0000 7fca2ba6b700 1 -- 172.21.15.72:0/3545243845 >> 172.21.15.72:0/3545243845 conn(0x7fca241005c0 msgr2=0x7fca24102a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:36.370 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.470+0000 7fca2ba6b700 1 -- 172.21.15.72:0/3545243845 shutdown_connections 2024-03-26T01:56:36.370 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.470+0000 7fca2ba6b700 1 -- 172.21.15.72:0/3545243845 wait complete. 2024-03-26T01:56:36.370 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.470+0000 7fca2ba6b700 1 Processor -- start 2024-03-26T01:56:36.370 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.470+0000 7fca2ba6b700 1 -- start start 2024-03-26T01:56:36.370 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.470+0000 7fca2ba6b700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca24105020 0x7fca24079bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:36.370 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.470+0000 7fca2ba6b700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca2419cbe0 con 0x7fca24105020 2024-03-26T01:56:36.370 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.470+0000 7fca29807700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca24105020 0x7fca24079bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:36.370 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.470+0000 7fca29807700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca24105020 0x7fca24079bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39846/0 (socket says 172.21.15.72:39846) 2024-03-26T01:56:36.370 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.470+0000 7fca29807700 1 -- 172.21.15.72:0/3064681661 learned_addr learned my addr 172.21.15.72:0/3064681661 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:36.370 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.471+0000 7fca29807700 1 -- 172.21.15.72:0/3064681661 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca20009550 con 0x7fca24105020 2024-03-26T01:56:36.370 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.471+0000 7fca29807700 1 --2- 172.21.15.72:0/3064681661 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca24105020 0x7fca24079bb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fca20000c00 tx=0x7fca20011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:36.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.471+0000 7fca1affd700 1 -- 172.21.15.72:0/3064681661 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca200036a0 con 0x7fca24105020 2024-03-26T01:56:36.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.471+0000 7fca2ba6b700 1 -- 172.21.15.72:0/3064681661 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca2419cd20 con 0x7fca24105020 2024-03-26T01:56:36.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.471+0000 7fca1affd700 1 -- 172.21.15.72:0/3064681661 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fca20024440 con 0x7fca24105020 2024-03-26T01:56:36.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.471+0000 7fca1affd700 1 -- 172.21.15.72:0/3064681661 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca2001a5c0 con 0x7fca24105020 2024-03-26T01:56:36.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.471+0000 7fca2ba6b700 1 -- 172.21.15.72:0/3064681661 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca2419d100 con 0x7fca24105020 2024-03-26T01:56:36.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.472+0000 7fca1affd700 1 -- 172.21.15.72:0/3064681661 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) v1 ==== 45949+0+0 (secure 0 0 0) 0x7fca2001a880 con 0x7fca24105020 2024-03-26T01:56:36.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.472+0000 7fca2ba6b700 1 -- 172.21.15.72:0/3064681661 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca24109450 con 0x7fca24105020 2024-03-26T01:56:36.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.472+0000 7fca1affd700 1 --2- 172.21.15.72:0/3064681661 >> v2:172.21.15.72:6800/2923394687 conn(0x7fca10038f90 0x7fca1003b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:36.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.472+0000 7fca1affd700 1 -- 172.21.15.72:0/3064681661 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7fca2004d010 con 0x7fca24105020 2024-03-26T01:56:36.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.472+0000 7fca29006700 1 --2- 172.21.15.72:0/3064681661 >> v2:172.21.15.72:6800/2923394687 conn(0x7fca10038f90 0x7fca1003b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:36.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.473+0000 7fca29006700 1 --2- 172.21.15.72:0/3064681661 >> v2:172.21.15.72:6800/2923394687 conn(0x7fca10038f90 0x7fca1003b450 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fca14006fd0 tx=0x7fca14006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:36.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.477+0000 7fca1affd700 1 -- 172.21.15.72:0/3064681661 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fca20022340 con 0x7fca24105020 2024-03-26T01:56:36.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.777+0000 7fca2ba6b700 1 -- 172.21.15.72:0/3064681661 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fca2419d440 con 0x7fca24105020 2024-03-26T01:56:36.371 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.779+0000 7fca1affd700 1 -- 172.21.15.72:0/3064681661 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7fca20018be0 con 0x7fca24105020 2024-03-26T01:56:36.372 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.781+0000 7fca2ba6b700 1 -- 172.21.15.72:0/3064681661 >> v2:172.21.15.72:6800/2923394687 conn(0x7fca10038f90 msgr2=0x7fca1003b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:36.372 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.781+0000 7fca2ba6b700 1 --2- 172.21.15.72:0/3064681661 >> v2:172.21.15.72:6800/2923394687 conn(0x7fca10038f90 0x7fca1003b450 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fca14006fd0 tx=0x7fca14006e40 comp rx=0 tx=0).stop 2024-03-26T01:56:36.372 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.781+0000 7fca2ba6b700 1 -- 172.21.15.72:0/3064681661 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca24105020 msgr2=0x7fca24079bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:36.372 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.781+0000 7fca2ba6b700 1 --2- 172.21.15.72:0/3064681661 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca24105020 0x7fca24079bb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fca20000c00 tx=0x7fca20011eb0 comp rx=0 tx=0).stop 2024-03-26T01:56:36.372 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.781+0000 7fca2ba6b700 1 -- 172.21.15.72:0/3064681661 shutdown_connections 2024-03-26T01:56:36.372 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.781+0000 7fca2ba6b700 1 --2- 172.21.15.72:0/3064681661 >> v2:172.21.15.72:6800/2923394687 conn(0x7fca10038f90 0x7fca1003b450 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:36.372 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.781+0000 7fca2ba6b700 1 --2- 172.21.15.72:0/3064681661 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fca24105020 0x7fca24079bb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:36.372 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.781+0000 7fca2ba6b700 1 -- 172.21.15.72:0/3064681661 >> 172.21.15.72:0/3064681661 conn(0x7fca241005c0 msgr2=0x7fca24101120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:36.372 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.781+0000 7fca2ba6b700 1 -- 172.21.15.72:0/3064681661 shutdown_connections 2024-03-26T01:56:36.372 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr 2024-03-26T01:56:35.781+0000 7fca2ba6b700 1 -- 172.21.15.72:0/3064681661 wait complete. 2024-03-26T01:56:36.372 INFO:teuthology.orchestra.run.smithi072.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-26T01:56:36.372 INFO:teuthology.orchestra.run.smithi072.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-26T01:56:36.372 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:56:36.372 INFO:teuthology.orchestra.run.smithi072.stdout: sudo /sbin/cephadm shell --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-26T01:56:36.373 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:56:36.373 INFO:teuthology.orchestra.run.smithi072.stdout:Or, if you are only running a single cluster on this host: 2024-03-26T01:56:36.373 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:56:36.373 INFO:teuthology.orchestra.run.smithi072.stdout: sudo /sbin/cephadm shell 2024-03-26T01:56:36.373 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:56:36.373 INFO:teuthology.orchestra.run.smithi072.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-26T01:56:36.373 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:56:36.373 INFO:teuthology.orchestra.run.smithi072.stdout: ceph telemetry on 2024-03-26T01:56:36.373 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:56:36.373 INFO:teuthology.orchestra.run.smithi072.stdout:For more information see: 2024-03-26T01:56:36.373 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:56:36.373 INFO:teuthology.orchestra.run.smithi072.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-26T01:56:36.373 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:56:36.373 INFO:teuthology.orchestra.run.smithi072.stdout:Bootstrap complete. 2024-03-26T01:56:36.411 INFO:tasks.cephadm:Fetching config... 2024-03-26T01:56:36.411 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-26T01:56:36.411 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-26T01:56:36.427 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-26T01:56:36.427 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-26T01:56:36.428 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-26T01:56:36.483 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-26T01:56:36.483 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-26T01:56:36.483 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-26T01:56:36.551 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-26T01:56:36.551 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-26T01:56:36.551 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-26T01:56:36.606 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-26T01:56:36.606 DEBUG:teuthology.orchestra.run.smithi072:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDafX/eCtR6yzLkLoyi6zM9XUNGF0VwQfVuDkeBWbODh4LOhGmzjkjDwh8crFLE0jViF8o5bZPNrj/Zls3czpgAV3LSNF1Oyckr0vKbH/Js6wXaTIp9b/EHarUwfO1s0h0ZoAAoSYjXJX/sNBERspEELeoRb5BbHrkHa/SdJA+L57skr9Hy6vXQ+yNjajzD3TlS6FYW3SwmWbXSl4euAEzQP2wd2NgobS3mwrhzyfI42L9YtPtnHkP/OObnBAdJtjAuJdHjq0xzUndFMVW1VBbpYLKHfBGZgzZie9lOL2/wLQGMRLRZf8KjwIZdD/Ui0yidfyppudiPdOyiT0f/KpGsLIRD9n5gQA99CKL786+bJqzFG8F2HhNZor5qx6cUY4jjJCCSeCEL1w/7hJJcV12/AKtG6/+mCMYjFE7kbj6/5ShSg5fY3RiG90sRTCRpGLPMsSmijkwO6p0gkswMFlUCIErUlAyW7LNowKDikihkuttvvEPfni2VzQqo5CZJns= ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T01:56:36.685 INFO:teuthology.orchestra.run.smithi072.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDafX/eCtR6yzLkLoyi6zM9XUNGF0VwQfVuDkeBWbODh4LOhGmzjkjDwh8crFLE0jViF8o5bZPNrj/Zls3czpgAV3LSNF1Oyckr0vKbH/Js6wXaTIp9b/EHarUwfO1s0h0ZoAAoSYjXJX/sNBERspEELeoRb5BbHrkHa/SdJA+L57skr9Hy6vXQ+yNjajzD3TlS6FYW3SwmWbXSl4euAEzQP2wd2NgobS3mwrhzyfI42L9YtPtnHkP/OObnBAdJtjAuJdHjq0xzUndFMVW1VBbpYLKHfBGZgzZie9lOL2/wLQGMRLRZf8KjwIZdD/Ui0yidfyppudiPdOyiT0f/KpGsLIRD9n5gQA99CKL786+bJqzFG8F2HhNZor5qx6cUY4jjJCCSeCEL1w/7hJJcV12/AKtG6/+mCMYjFE7kbj6/5ShSg5fY3RiG90sRTCRpGLPMsSmijkwO6p0gkswMFlUCIErUlAyW7LNowKDikihkuttvvEPfni2VzQqo5CZJns= ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T01:56:36.700 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDafX/eCtR6yzLkLoyi6zM9XUNGF0VwQfVuDkeBWbODh4LOhGmzjkjDwh8crFLE0jViF8o5bZPNrj/Zls3czpgAV3LSNF1Oyckr0vKbH/Js6wXaTIp9b/EHarUwfO1s0h0ZoAAoSYjXJX/sNBERspEELeoRb5BbHrkHa/SdJA+L57skr9Hy6vXQ+yNjajzD3TlS6FYW3SwmWbXSl4euAEzQP2wd2NgobS3mwrhzyfI42L9YtPtnHkP/OObnBAdJtjAuJdHjq0xzUndFMVW1VBbpYLKHfBGZgzZie9lOL2/wLQGMRLRZf8KjwIZdD/Ui0yidfyppudiPdOyiT0f/KpGsLIRD9n5gQA99CKL786+bJqzFG8F2HhNZor5qx6cUY4jjJCCSeCEL1w/7hJJcV12/AKtG6/+mCMYjFE7kbj6/5ShSg5fY3RiG90sRTCRpGLPMsSmijkwO6p0gkswMFlUCIErUlAyW7LNowKDikihkuttvvEPfni2VzQqo5CZJns= ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T01:56:36.746 INFO:teuthology.orchestra.run.smithi173.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDafX/eCtR6yzLkLoyi6zM9XUNGF0VwQfVuDkeBWbODh4LOhGmzjkjDwh8crFLE0jViF8o5bZPNrj/Zls3czpgAV3LSNF1Oyckr0vKbH/Js6wXaTIp9b/EHarUwfO1s0h0ZoAAoSYjXJX/sNBERspEELeoRb5BbHrkHa/SdJA+L57skr9Hy6vXQ+yNjajzD3TlS6FYW3SwmWbXSl4euAEzQP2wd2NgobS3mwrhzyfI42L9YtPtnHkP/OObnBAdJtjAuJdHjq0xzUndFMVW1VBbpYLKHfBGZgzZie9lOL2/wLQGMRLRZf8KjwIZdD/Ui0yidfyppudiPdOyiT0f/KpGsLIRD9n5gQA99CKL786+bJqzFG8F2HhNZor5qx6cUY4jjJCCSeCEL1w/7hJJcV12/AKtG6/+mCMYjFE7kbj6/5ShSg5fY3RiG90sRTCRpGLPMsSmijkwO6p0gkswMFlUCIErUlAyW7LNowKDikihkuttvvEPfni2VzQqo5CZJns= ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T01:56:36.760 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-26T01:56:36.918 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:36 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3064681661' entity='client.admin' 2024-03-26T01:56:37.007 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:56:39.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.472+0000 7fab1ba29700 1 -- 172.21.15.72:0/3280676096 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab141009b0 msgr2=0x7fab14100dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:39.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.472+0000 7fab1ba29700 1 --2- 172.21.15.72:0/3280676096 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab141009b0 0x7fab14100dd0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fab04009910 tx=0x7fab04009c20 comp rx=0 tx=0).stop 2024-03-26T01:56:39.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.472+0000 7fab1ba29700 1 -- 172.21.15.72:0/3280676096 shutdown_connections 2024-03-26T01:56:39.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.472+0000 7fab1ba29700 1 --2- 172.21.15.72:0/3280676096 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab141009b0 0x7fab14100dd0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:39.478 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.472+0000 7fab1ba29700 1 -- 172.21.15.72:0/3280676096 >> 172.21.15.72:0/3280676096 conn(0x7fab140fbf50 msgr2=0x7fab140fe390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:39.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.472+0000 7fab1ba29700 1 -- 172.21.15.72:0/3280676096 shutdown_connections 2024-03-26T01:56:39.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.472+0000 7fab1ba29700 1 -- 172.21.15.72:0/3280676096 wait complete. 2024-03-26T01:56:39.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.473+0000 7fab1ba29700 1 Processor -- start 2024-03-26T01:56:39.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.473+0000 7fab1ba29700 1 -- start start 2024-03-26T01:56:39.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.473+0000 7fab1ba29700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab141009b0 0x7fab14069f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:39.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.473+0000 7fab1ba29700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab141993b0 con 0x7fab141009b0 2024-03-26T01:56:39.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.473+0000 7fab197c5700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab141009b0 0x7fab14069f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:39.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.473+0000 7fab197c5700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab141009b0 0x7fab14069f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:39854/0 (socket says 172.21.15.72:39854) 2024-03-26T01:56:39.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.473+0000 7fab197c5700 1 -- 172.21.15.72:0/1005795215 learned_addr learned my addr 172.21.15.72:0/1005795215 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:39.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.473+0000 7fab197c5700 1 -- 172.21.15.72:0/1005795215 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab040095f0 con 0x7fab141009b0 2024-03-26T01:56:39.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.474+0000 7fab197c5700 1 --2- 172.21.15.72:0/1005795215 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab141009b0 0x7fab14069f50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fab040057a0 tx=0x7fab04004f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:39.479 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.474+0000 7fab0a7fc700 1 -- 172.21.15.72:0/1005795215 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fab0401d070 con 0x7fab141009b0 2024-03-26T01:56:39.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.474+0000 7fab0a7fc700 1 -- 172.21.15.72:0/1005795215 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fab04022470 con 0x7fab141009b0 2024-03-26T01:56:39.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.474+0000 7fab1ba29700 1 -- 172.21.15.72:0/1005795215 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab141994f0 con 0x7fab141009b0 2024-03-26T01:56:39.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.474+0000 7fab0a7fc700 1 -- 172.21.15.72:0/1005795215 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fab0400f5f0 con 0x7fab141009b0 2024-03-26T01:56:39.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.474+0000 7fab1ba29700 1 -- 172.21.15.72:0/1005795215 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab141998d0 con 0x7fab141009b0 2024-03-26T01:56:39.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.475+0000 7fab0a7fc700 1 -- 172.21.15.72:0/1005795215 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) v1 ==== 45949+0+0 (secure 0 0 0) 0x7fab0400ba90 con 0x7fab141009b0 2024-03-26T01:56:39.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.475+0000 7fab1ba29700 1 -- 172.21.15.72:0/1005795215 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab1418f510 con 0x7fab141009b0 2024-03-26T01:56:39.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.475+0000 7fab0a7fc700 1 --2- 172.21.15.72:0/1005795215 >> v2:172.21.15.72:6800/2923394687 conn(0x7fab00038f40 0x7fab0003b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:39.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.475+0000 7fab0a7fc700 1 -- 172.21.15.72:0/1005795215 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7fab0404d280 con 0x7fab141009b0 2024-03-26T01:56:39.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.475+0000 7fab18fc4700 1 --2- 172.21.15.72:0/1005795215 >> v2:172.21.15.72:6800/2923394687 conn(0x7fab00038f40 0x7fab0003b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:39.480 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.475+0000 7fab18fc4700 1 --2- 172.21.15.72:0/1005795215 >> v2:172.21.15.72:6800/2923394687 conn(0x7fab00038f40 0x7fab0003b400 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fab10006fd0 tx=0x7fab10006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:39.481 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.480+0000 7fab0a7fc700 1 -- 172.21.15.72:0/1005795215 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fab040228c0 con 0x7fab141009b0 2024-03-26T01:56:39.571 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:39 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:39.571 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:39 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:39.571 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:39 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi072", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:56:39.571 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:39 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:39.571 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:39 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:39.571 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:39 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:39.681 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.680+0000 7fab1ba29700 1 -- 172.21.15.72:0/1005795215 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fab141061d0 con 0x7fab141009b0 2024-03-26T01:56:39.686 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.685+0000 7fab0a7fc700 1 -- 172.21.15.72:0/1005795215 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fab141061d0 con 0x7fab141009b0 2024-03-26T01:56:39.690 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.688+0000 7fab1ba29700 1 -- 172.21.15.72:0/1005795215 >> v2:172.21.15.72:6800/2923394687 conn(0x7fab00038f40 msgr2=0x7fab0003b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:39.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.688+0000 7fab1ba29700 1 --2- 172.21.15.72:0/1005795215 >> v2:172.21.15.72:6800/2923394687 conn(0x7fab00038f40 0x7fab0003b400 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fab10006fd0 tx=0x7fab10006e40 comp rx=0 tx=0).stop 2024-03-26T01:56:39.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.688+0000 7fab1ba29700 1 -- 172.21.15.72:0/1005795215 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab141009b0 msgr2=0x7fab14069f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:39.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.688+0000 7fab1ba29700 1 --2- 172.21.15.72:0/1005795215 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab141009b0 0x7fab14069f50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fab040057a0 tx=0x7fab04004f20 comp rx=0 tx=0).stop 2024-03-26T01:56:39.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.688+0000 7fab1ba29700 1 -- 172.21.15.72:0/1005795215 shutdown_connections 2024-03-26T01:56:39.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.688+0000 7fab1ba29700 1 --2- 172.21.15.72:0/1005795215 >> v2:172.21.15.72:6800/2923394687 conn(0x7fab00038f40 0x7fab0003b400 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:39.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.688+0000 7fab1ba29700 1 --2- 172.21.15.72:0/1005795215 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fab141009b0 0x7fab14069f50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:39.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.688+0000 7fab1ba29700 1 -- 172.21.15.72:0/1005795215 >> 172.21.15.72:0/1005795215 conn(0x7fab140fbf50 msgr2=0x7fab140fcc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:39.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.688+0000 7fab1ba29700 1 -- 172.21.15.72:0/1005795215 shutdown_connections 2024-03-26T01:56:39.691 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:39.688+0000 7fab1ba29700 1 -- 172.21.15.72:0/1005795215 wait complete. 2024-03-26T01:56:40.348 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-26T01:56:40.348 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-26T01:56:40.914 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1005795215' entity='client.admin' 2024-03-26T01:56:40.914 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:40 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:40.914 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:40 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:41.014 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:56:42.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.502+0000 7efdbf40a700 1 -- 172.21.15.72:0/560604187 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efdb80ffda0 msgr2=0x7efdb81001c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:42.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.502+0000 7efdbf40a700 1 --2- 172.21.15.72:0/560604187 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efdb80ffda0 0x7efdb81001c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7efdb4009910 tx=0x7efdb4009c20 comp rx=0 tx=0).stop 2024-03-26T01:56:42.504 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.502+0000 7efdbf40a700 1 -- 172.21.15.72:0/560604187 shutdown_connections 2024-03-26T01:56:42.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.502+0000 7efdbf40a700 1 --2- 172.21.15.72:0/560604187 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efdb80ffda0 0x7efdb81001c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:42.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.502+0000 7efdbf40a700 1 -- 172.21.15.72:0/560604187 >> 172.21.15.72:0/560604187 conn(0x7efdb80fb6e0 msgr2=0x7efdb80fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:42.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.502+0000 7efdbf40a700 1 -- 172.21.15.72:0/560604187 shutdown_connections 2024-03-26T01:56:42.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.502+0000 7efdbf40a700 1 -- 172.21.15.72:0/560604187 wait complete. 2024-03-26T01:56:42.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.502+0000 7efdbf40a700 1 Processor -- start 2024-03-26T01:56:42.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.502+0000 7efdbf40a700 1 -- start start 2024-03-26T01:56:42.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.502+0000 7efdbf40a700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efdb80ffda0 0x7efdb81026a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:42.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.502+0000 7efdbf40a700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdb8102be0 con 0x7efdb80ffda0 2024-03-26T01:56:42.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.503+0000 7efdbd1a6700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efdb80ffda0 0x7efdb81026a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:42.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.503+0000 7efdbd1a6700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efdb80ffda0 0x7efdb81026a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44868/0 (socket says 172.21.15.72:44868) 2024-03-26T01:56:42.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.503+0000 7efdbd1a6700 1 -- 172.21.15.72:0/2607193333 learned_addr learned my addr 172.21.15.72:0/2607193333 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:42.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.503+0000 7efdbd1a6700 1 -- 172.21.15.72:0/2607193333 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdb40095f0 con 0x7efdb80ffda0 2024-03-26T01:56:42.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.503+0000 7efdbd1a6700 1 --2- 172.21.15.72:0/2607193333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efdb80ffda0 0x7efdb81026a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7efdb40057a0 tx=0x7efdb4004f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:42.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.503+0000 7efdae7fc700 1 -- 172.21.15.72:0/2607193333 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efdb401d070 con 0x7efdb80ffda0 2024-03-26T01:56:42.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.503+0000 7efdae7fc700 1 -- 172.21.15.72:0/2607193333 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7efdb4022470 con 0x7efdb80ffda0 2024-03-26T01:56:42.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.503+0000 7efdbf40a700 1 -- 172.21.15.72:0/2607193333 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdb8100dd0 con 0x7efdb80ffda0 2024-03-26T01:56:42.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.503+0000 7efdae7fc700 1 -- 172.21.15.72:0/2607193333 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efdb400f5f0 con 0x7efdb80ffda0 2024-03-26T01:56:42.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.503+0000 7efdbf40a700 1 -- 172.21.15.72:0/2607193333 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdb81012d0 con 0x7efdb80ffda0 2024-03-26T01:56:42.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.504+0000 7efdae7fc700 1 -- 172.21.15.72:0/2607193333 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) v1 ==== 45949+0+0 (secure 0 0 0) 0x7efdb400f830 con 0x7efdb80ffda0 2024-03-26T01:56:42.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.504+0000 7efdbf40a700 1 -- 172.21.15.72:0/2607193333 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdb81078a0 con 0x7efdb80ffda0 2024-03-26T01:56:42.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.504+0000 7efdae7fc700 1 --2- 172.21.15.72:0/2607193333 >> v2:172.21.15.72:6800/2923394687 conn(0x7efda4038f40 0x7efda403b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:42.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.504+0000 7efdae7fc700 1 -- 172.21.15.72:0/2607193333 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7efdb404d1c0 con 0x7efdb80ffda0 2024-03-26T01:56:42.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.504+0000 7efdbc9a5700 1 --2- 172.21.15.72:0/2607193333 >> v2:172.21.15.72:6800/2923394687 conn(0x7efda4038f40 0x7efda403b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:42.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.505+0000 7efdbc9a5700 1 --2- 172.21.15.72:0/2607193333 >> v2:172.21.15.72:6800/2923394687 conn(0x7efda4038f40 0x7efda403b400 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7efda8006fd0 tx=0x7efda8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:42.510 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.509+0000 7efdae7fc700 1 -- 172.21.15.72:0/2607193333 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7efdb400fe60 con 0x7efdb80ffda0 2024-03-26T01:56:42.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.711+0000 7efdbf40a700 1 -- 172.21.15.72:0/2607193333 --> v2:172.21.15.72:6800/2923394687 -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7efdb8107220 con 0x7efda4038f40 2024-03-26T01:56:42.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.714+0000 7efdae7fc700 1 -- 172.21.15.72:0/2607193333 <== mgr.14152 v2:172.21.15.72:6800/2923394687 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7efdb8107220 con 0x7efda4038f40 2024-03-26T01:56:42.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.716+0000 7efdbf40a700 1 -- 172.21.15.72:0/2607193333 >> v2:172.21.15.72:6800/2923394687 conn(0x7efda4038f40 msgr2=0x7efda403b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:42.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.716+0000 7efdbf40a700 1 --2- 172.21.15.72:0/2607193333 >> v2:172.21.15.72:6800/2923394687 conn(0x7efda4038f40 0x7efda403b400 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7efda8006fd0 tx=0x7efda8006e40 comp rx=0 tx=0).stop 2024-03-26T01:56:42.718 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.716+0000 7efdbf40a700 1 -- 172.21.15.72:0/2607193333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efdb80ffda0 msgr2=0x7efdb81026a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:42.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.716+0000 7efdbf40a700 1 --2- 172.21.15.72:0/2607193333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efdb80ffda0 0x7efdb81026a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7efdb40057a0 tx=0x7efdb4004f20 comp rx=0 tx=0).stop 2024-03-26T01:56:42.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.717+0000 7efdbf40a700 1 -- 172.21.15.72:0/2607193333 shutdown_connections 2024-03-26T01:56:42.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.717+0000 7efdbf40a700 1 --2- 172.21.15.72:0/2607193333 >> v2:172.21.15.72:6800/2923394687 conn(0x7efda4038f40 0x7efda403b400 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:42.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.717+0000 7efdbf40a700 1 --2- 172.21.15.72:0/2607193333 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efdb80ffda0 0x7efdb81026a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:42.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.717+0000 7efdbf40a700 1 -- 172.21.15.72:0/2607193333 >> 172.21.15.72:0/2607193333 conn(0x7efdb80fb6e0 msgr2=0x7efdb80fc3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:42.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.717+0000 7efdbf40a700 1 -- 172.21.15.72:0/2607193333 shutdown_connections 2024-03-26T01:56:42.719 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:42.717+0000 7efdbf40a700 1 -- 172.21.15.72:0/2607193333 wait complete. 2024-03-26T01:56:43.462 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi173 2024-03-26T01:56:43.462 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-26T01:56:43.462 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.conf 2024-03-26T01:56:43.480 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-26T01:56:43.480 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:56:43.539 INFO:tasks.cephadm:Adding host smithi173 to orchestrator... 2024-03-26T01:56:43.539 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph orch host add smithi173 2024-03-26T01:56:43.777 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:56:43.968 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:43 smithi072 ceph-mon[30177]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:43.968 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:43 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:43.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:43 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:43.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:43 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:43.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:43 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:56:43.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:43 smithi072 ceph-mon[30177]: Updating smithi072:/etc/ceph/ceph.conf 2024-03-26T01:56:43.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:43 smithi072 ceph-mon[30177]: Updating smithi072:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:56:43.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:43 smithi072 ceph-mon[30177]: Updating smithi072:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:56:43.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:43 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:43.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:43 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:43.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:43 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:44.831 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:44 smithi072 ceph-mon[30177]: Updating smithi072:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:56:45.346 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.344+0000 7fa7f2d49700 1 -- 172.21.15.72:0/2766894427 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7ec102c50 msgr2=0x7fa7ec103030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:45.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.344+0000 7fa7f2d49700 1 --2- 172.21.15.72:0/2766894427 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7ec102c50 0x7fa7ec103030 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0009910 tx=0x7fa7e0009c20 comp rx=0 tx=0).stop 2024-03-26T01:56:45.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.344+0000 7fa7f2d49700 1 -- 172.21.15.72:0/2766894427 shutdown_connections 2024-03-26T01:56:45.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.344+0000 7fa7f2d49700 1 --2- 172.21.15.72:0/2766894427 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7ec102c50 0x7fa7ec103030 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:45.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.344+0000 7fa7f2d49700 1 -- 172.21.15.72:0/2766894427 >> 172.21.15.72:0/2766894427 conn(0x7fa7ec0fc4c0 msgr2=0x7fa7ec0fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:45.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.344+0000 7fa7f2d49700 1 -- 172.21.15.72:0/2766894427 shutdown_connections 2024-03-26T01:56:45.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.344+0000 7fa7f2d49700 1 -- 172.21.15.72:0/2766894427 wait complete. 2024-03-26T01:56:45.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.344+0000 7fa7f2d49700 1 Processor -- start 2024-03-26T01:56:45.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.344+0000 7fa7f2d49700 1 -- start start 2024-03-26T01:56:45.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.344+0000 7fa7f2d49700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7ec102c50 0x7fa7ec19c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:45.347 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.344+0000 7fa7f2d49700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec19ccd0 con 0x7fa7ec102c50 2024-03-26T01:56:45.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.344+0000 7fa7f0ae5700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7ec102c50 0x7fa7ec19c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:45.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.345+0000 7fa7f0ae5700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7ec102c50 0x7fa7ec19c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:44884/0 (socket says 172.21.15.72:44884) 2024-03-26T01:56:45.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.345+0000 7fa7f0ae5700 1 -- 172.21.15.72:0/890136983 learned_addr learned my addr 172.21.15.72:0/890136983 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:45.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.345+0000 7fa7f0ae5700 1 -- 172.21.15.72:0/890136983 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e00095f0 con 0x7fa7ec102c50 2024-03-26T01:56:45.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.345+0000 7fa7f0ae5700 1 --2- 172.21.15.72:0/890136983 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7ec102c50 0x7fa7ec19c790 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa7e000f690 tx=0x7fa7e000f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:45.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.345+0000 7fa7e9ffb700 1 -- 172.21.15.72:0/890136983 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7e001d070 con 0x7fa7ec102c50 2024-03-26T01:56:45.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.345+0000 7fa7e9ffb700 1 -- 172.21.15.72:0/890136983 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fa7e0004500 con 0x7fa7ec102c50 2024-03-26T01:56:45.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.345+0000 7fa7f2d49700 1 -- 172.21.15.72:0/890136983 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7ec19ced0 con 0x7fa7ec102c50 2024-03-26T01:56:45.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.345+0000 7fa7e9ffb700 1 -- 172.21.15.72:0/890136983 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7e0003c10 con 0x7fa7ec102c50 2024-03-26T01:56:45.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.345+0000 7fa7f2d49700 1 -- 172.21.15.72:0/890136983 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7ec19d2f0 con 0x7fa7ec102c50 2024-03-26T01:56:45.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.346+0000 7fa7e9ffb700 1 -- 172.21.15.72:0/890136983 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) v1 ==== 45949+0+0 (secure 0 0 0) 0x7fa7e0004050 con 0x7fa7ec102c50 2024-03-26T01:56:45.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.346+0000 7fa7f2d49700 1 -- 172.21.15.72:0/890136983 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7ec0636b0 con 0x7fa7ec102c50 2024-03-26T01:56:45.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.346+0000 7fa7e9ffb700 1 --2- 172.21.15.72:0/890136983 >> v2:172.21.15.72:6800/2923394687 conn(0x7fa7dc038f90 0x7fa7dc03b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:45.348 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.346+0000 7fa7e9ffb700 1 -- 172.21.15.72:0/890136983 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7fa7e004cc90 con 0x7fa7ec102c50 2024-03-26T01:56:45.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.346+0000 7fa7ebfff700 1 --2- 172.21.15.72:0/890136983 >> v2:172.21.15.72:6800/2923394687 conn(0x7fa7dc038f90 0x7fa7dc03b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:45.349 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.347+0000 7fa7ebfff700 1 --2- 172.21.15.72:0/890136983 >> v2:172.21.15.72:6800/2923394687 conn(0x7fa7dc038f90 0x7fa7dc03b450 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8006fd0 tx=0x7fa7d8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:45.351 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.351+0000 7fa7e9ffb700 1 -- 172.21.15.72:0/890136983 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa7e0015530 con 0x7fa7ec102c50 2024-03-26T01:56:45.546 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:45.546+0000 7fa7f2d49700 1 -- 172.21.15.72:0/890136983 --> v2:172.21.15.72:6800/2923394687 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7ec10b670 con 0x7fa7dc038f90 2024-03-26T01:56:46.718 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:46 smithi072 ceph-mon[30177]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:46.718 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:46 smithi072 ceph-mon[30177]: Deploying cephadm binary to smithi173 2024-03-26T01:56:48.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:48 smithi072 ceph-mon[30177]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:49.797 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:49.797+0000 7fa7e9ffb700 1 -- 172.21.15.72:0/890136983 <== mgr.14152 v2:172.21.15.72:6800/2923394687 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fa7ec10b670 con 0x7fa7dc038f90 2024-03-26T01:56:49.798 INFO:teuthology.orchestra.run.smithi072.stdout:Added host 'smithi173' with addr '172.21.15.173' 2024-03-26T01:56:49.799 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:49.799+0000 7fa7f2d49700 1 -- 172.21.15.72:0/890136983 >> v2:172.21.15.72:6800/2923394687 conn(0x7fa7dc038f90 msgr2=0x7fa7dc03b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:49.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:49.799+0000 7fa7f2d49700 1 --2- 172.21.15.72:0/890136983 >> v2:172.21.15.72:6800/2923394687 conn(0x7fa7dc038f90 0x7fa7dc03b450 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa7d8006fd0 tx=0x7fa7d8006e40 comp rx=0 tx=0).stop 2024-03-26T01:56:49.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:49.799+0000 7fa7f2d49700 1 -- 172.21.15.72:0/890136983 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7ec102c50 msgr2=0x7fa7ec19c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:49.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:49.799+0000 7fa7f2d49700 1 --2- 172.21.15.72:0/890136983 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7ec102c50 0x7fa7ec19c790 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa7e000f690 tx=0x7fa7e000f770 comp rx=0 tx=0).stop 2024-03-26T01:56:49.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:49.799+0000 7fa7f2d49700 1 -- 172.21.15.72:0/890136983 shutdown_connections 2024-03-26T01:56:49.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:49.799+0000 7fa7f2d49700 1 --2- 172.21.15.72:0/890136983 >> v2:172.21.15.72:6800/2923394687 conn(0x7fa7dc038f90 0x7fa7dc03b450 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:49.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:49.799+0000 7fa7f2d49700 1 --2- 172.21.15.72:0/890136983 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa7ec102c50 0x7fa7ec19c790 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:49.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:49.799+0000 7fa7f2d49700 1 -- 172.21.15.72:0/890136983 >> 172.21.15.72:0/890136983 conn(0x7fa7ec0fc4c0 msgr2=0x7fa7ec0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:49.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:49.800+0000 7fa7f2d49700 1 -- 172.21.15.72:0/890136983 shutdown_connections 2024-03-26T01:56:49.800 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:49.800+0000 7fa7f2d49700 1 -- 172.21.15.72:0/890136983 wait complete. 2024-03-26T01:56:50.455 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:50 smithi072 ceph-mon[30177]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:50.456 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:50 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:50.456 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:50 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:50.490 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-26T01:56:50.729 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:56:51.179 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:51 smithi072 ceph-mon[30177]: Added host smithi173 2024-03-26T01:56:51.179 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:51 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:51.179 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:51 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:52.218 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:52 smithi072 ceph-mon[30177]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:52.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.263+0000 7f76c5622700 1 -- 172.21.15.72:0/3981437961 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76c0106950 msgr2=0x7f76c0106d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:52.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.263+0000 7f76c5622700 1 --2- 172.21.15.72:0/3981437961 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76c0106950 0x7f76c0106d30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f76b0009910 tx=0x7f76b0009c20 comp rx=0 tx=0).stop 2024-03-26T01:56:52.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.263+0000 7f76c5622700 1 -- 172.21.15.72:0/3981437961 shutdown_connections 2024-03-26T01:56:52.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.263+0000 7f76c5622700 1 --2- 172.21.15.72:0/3981437961 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76c0106950 0x7f76c0106d30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:52.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.263+0000 7f76c5622700 1 -- 172.21.15.72:0/3981437961 >> 172.21.15.72:0/3981437961 conn(0x7f76c00775c0 msgr2=0x7f76c00fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:52.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.263+0000 7f76c5622700 1 -- 172.21.15.72:0/3981437961 shutdown_connections 2024-03-26T01:56:52.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.263+0000 7f76c5622700 1 -- 172.21.15.72:0/3981437961 wait complete. 2024-03-26T01:56:52.264 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.264+0000 7f76c5622700 1 Processor -- start 2024-03-26T01:56:52.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.264+0000 7f76c5622700 1 -- start start 2024-03-26T01:56:52.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.264+0000 7f76c5622700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76c0106950 0x7f76c0197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:52.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.264+0000 7f76c5622700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c006a3f0 con 0x7f76c0106950 2024-03-26T01:56:52.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.264+0000 7f76beffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76c0106950 0x7f76c0197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:52.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.264+0000 7f76beffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76c0106950 0x7f76c0197940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40602/0 (socket says 172.21.15.72:40602) 2024-03-26T01:56:52.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.264+0000 7f76beffd700 1 -- 172.21.15.72:0/2119268525 learned_addr learned my addr 172.21.15.72:0/2119268525 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:52.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.265+0000 7f76beffd700 1 -- 172.21.15.72:0/2119268525 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b00095f0 con 0x7f76c0106950 2024-03-26T01:56:52.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.265+0000 7f76beffd700 1 --2- 172.21.15.72:0/2119268525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76c0106950 0x7f76c0197940 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f76b000f690 tx=0x7f76b000f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:52.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.265+0000 7f76a7fff700 1 -- 172.21.15.72:0/2119268525 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f76b001d070 con 0x7f76c0106950 2024-03-26T01:56:52.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.265+0000 7f76a7fff700 1 -- 172.21.15.72:0/2119268525 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f76b0004500 con 0x7f76c0106950 2024-03-26T01:56:52.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.265+0000 7f76a7fff700 1 -- 172.21.15.72:0/2119268525 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f76b0003b30 con 0x7f76c0106950 2024-03-26T01:56:52.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.265+0000 7f76c5622700 1 -- 172.21.15.72:0/2119268525 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76c0197f40 con 0x7f76c0106950 2024-03-26T01:56:52.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.265+0000 7f76c5622700 1 -- 172.21.15.72:0/2119268525 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76c01983c0 con 0x7f76c0106950 2024-03-26T01:56:52.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.266+0000 7f76a7fff700 1 -- 172.21.15.72:0/2119268525 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) v1 ==== 45949+0+0 (secure 0 0 0) 0x7f76b002b440 con 0x7f76c0106950 2024-03-26T01:56:52.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.266+0000 7f76c5622700 1 -- 172.21.15.72:0/2119268525 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76c0104c00 con 0x7f76c0106950 2024-03-26T01:56:52.267 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.266+0000 7f76a7fff700 1 --2- 172.21.15.72:0/2119268525 >> v2:172.21.15.72:6800/2923394687 conn(0x7f76a8038f40 0x7f76a803b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:52.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.266+0000 7f76a7fff700 1 -- 172.21.15.72:0/2119268525 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7f76b004cdb0 con 0x7f76c0106950 2024-03-26T01:56:52.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.266+0000 7f76be7fc700 1 --2- 172.21.15.72:0/2119268525 >> v2:172.21.15.72:6800/2923394687 conn(0x7f76a8038f40 0x7f76a803b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:52.268 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.266+0000 7f76be7fc700 1 --2- 172.21.15.72:0/2119268525 >> v2:172.21.15.72:6800/2923394687 conn(0x7f76a8038f40 0x7f76a803b400 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f76b4006fd0 tx=0x7f76b4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:52.271 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.271+0000 7f76a7fff700 1 -- 172.21.15.72:0/2119268525 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f76b0015350 con 0x7f76c0106950 2024-03-26T01:56:52.473 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.472+0000 7f76c5622700 1 -- 172.21.15.72:0/2119268525 --> v2:172.21.15.72:6800/2923394687 -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f76c0069aa0 con 0x7f76a8038f40 2024-03-26T01:56:52.474 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.474+0000 7f76a7fff700 1 -- 172.21.15.72:0/2119268525 <== mgr.14152 v2:172.21.15.72:6800/2923394687 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f76c0069aa0 con 0x7f76a8038f40 2024-03-26T01:56:52.474 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T01:56:52.474 INFO:teuthology.orchestra.run.smithi072.stdout:[{"addr": "172.21.15.72", "hostname": "smithi072", "labels": [], "status": ""}, {"addr": "172.21.15.173", "hostname": "smithi173", "labels": [], "status": ""}] 2024-03-26T01:56:52.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.476+0000 7f76c5622700 1 -- 172.21.15.72:0/2119268525 >> v2:172.21.15.72:6800/2923394687 conn(0x7f76a8038f40 msgr2=0x7f76a803b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:52.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.476+0000 7f76c5622700 1 --2- 172.21.15.72:0/2119268525 >> v2:172.21.15.72:6800/2923394687 conn(0x7f76a8038f40 0x7f76a803b400 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f76b4006fd0 tx=0x7f76b4006e40 comp rx=0 tx=0).stop 2024-03-26T01:56:52.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.476+0000 7f76c5622700 1 -- 172.21.15.72:0/2119268525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76c0106950 msgr2=0x7f76c0197940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:52.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.476+0000 7f76c5622700 1 --2- 172.21.15.72:0/2119268525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76c0106950 0x7f76c0197940 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f76b000f690 tx=0x7f76b000f770 comp rx=0 tx=0).stop 2024-03-26T01:56:52.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.476+0000 7f76c5622700 1 -- 172.21.15.72:0/2119268525 shutdown_connections 2024-03-26T01:56:52.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.476+0000 7f76c5622700 1 --2- 172.21.15.72:0/2119268525 >> v2:172.21.15.72:6800/2923394687 conn(0x7f76a8038f40 0x7f76a803b400 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:52.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.476+0000 7f76c5622700 1 --2- 172.21.15.72:0/2119268525 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76c0106950 0x7f76c0197940 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:52.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.476+0000 7f76c5622700 1 -- 172.21.15.72:0/2119268525 >> 172.21.15.72:0/2119268525 conn(0x7f76c00775c0 msgr2=0x7f76c0108690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:52.476 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.476+0000 7f76c5622700 1 -- 172.21.15.72:0/2119268525 shutdown_connections 2024-03-26T01:56:52.477 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:52.476+0000 7f76c5622700 1 -- 172.21.15.72:0/2119268525 wait complete. 2024-03-26T01:56:53.071 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-26T01:56:53.071 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-26T01:56:53.302 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:56:53.641 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:53 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:53.641 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:53 smithi072 ceph-mon[30177]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T01:56:53.641 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:53 smithi072 ceph-mon[30177]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:54.825 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.824+0000 7f3371d75700 1 -- 172.21.15.72:0/706800175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f336c106950 msgr2=0x7f336c106d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:54.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.824+0000 7f3371d75700 1 --2- 172.21.15.72:0/706800175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f336c106950 0x7f336c106d30 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f335c009910 tx=0x7f335c009c20 comp rx=0 tx=0).stop 2024-03-26T01:56:54.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.824+0000 7f3371d75700 1 -- 172.21.15.72:0/706800175 shutdown_connections 2024-03-26T01:56:54.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.824+0000 7f3371d75700 1 --2- 172.21.15.72:0/706800175 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f336c106950 0x7f336c106d30 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:54.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.824+0000 7f3371d75700 1 -- 172.21.15.72:0/706800175 >> 172.21.15.72:0/706800175 conn(0x7f336c0775c0 msgr2=0x7f336c0fe890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:54.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.824+0000 7f3371d75700 1 -- 172.21.15.72:0/706800175 shutdown_connections 2024-03-26T01:56:54.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.824+0000 7f3371d75700 1 -- 172.21.15.72:0/706800175 wait complete. 2024-03-26T01:56:54.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.824+0000 7f3371d75700 1 Processor -- start 2024-03-26T01:56:54.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.825+0000 7f3371d75700 1 -- start start 2024-03-26T01:56:54.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.825+0000 7f3371d75700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f336c106950 0x7f336c102520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:54.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.825+0000 7f3371d75700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336c1043c0 con 0x7f336c106950 2024-03-26T01:56:54.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.825+0000 7f336b7fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f336c106950 0x7f336c102520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:54.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.825+0000 7f336b7fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f336c106950 0x7f336c102520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40612/0 (socket says 172.21.15.72:40612) 2024-03-26T01:56:54.826 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.825+0000 7f336b7fe700 1 -- 172.21.15.72:0/2963326343 learned_addr learned my addr 172.21.15.72:0/2963326343 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:56:54.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.825+0000 7f336b7fe700 1 -- 172.21.15.72:0/2963326343 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f335c0095f0 con 0x7f336c106950 2024-03-26T01:56:54.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.825+0000 7f336b7fe700 1 --2- 172.21.15.72:0/2963326343 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f336c106950 0x7f336c102520 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f335c00f690 tx=0x7f335c00f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:54.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.825+0000 7f3368ff9700 1 -- 172.21.15.72:0/2963326343 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f335c01d070 con 0x7f336c106950 2024-03-26T01:56:54.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.825+0000 7f3368ff9700 1 -- 172.21.15.72:0/2963326343 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f335c004500 con 0x7f336c106950 2024-03-26T01:56:54.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.825+0000 7f3371d75700 1 -- 172.21.15.72:0/2963326343 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f336c102a60 con 0x7f336c106950 2024-03-26T01:56:54.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.825+0000 7f3368ff9700 1 -- 172.21.15.72:0/2963326343 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f335c003b30 con 0x7f336c106950 2024-03-26T01:56:54.827 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.825+0000 7f3371d75700 1 -- 172.21.15.72:0/2963326343 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f336c102ee0 con 0x7f336c106950 2024-03-26T01:56:54.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.826+0000 7f3368ff9700 1 -- 172.21.15.72:0/2963326343 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) v1 ==== 45949+0+0 (secure 0 0 0) 0x7f335c004050 con 0x7f336c106950 2024-03-26T01:56:54.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.826+0000 7f3371d75700 1 -- 172.21.15.72:0/2963326343 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f336c0636b0 con 0x7f336c106950 2024-03-26T01:56:54.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.827+0000 7f3368ff9700 1 --2- 172.21.15.72:0/2963326343 >> v2:172.21.15.72:6800/2923394687 conn(0x7f3354038f40 0x7f335403b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:54.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.827+0000 7f3368ff9700 1 -- 172.21.15.72:0/2963326343 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (secure 0 0 0) 0x7f335c04cc10 con 0x7f336c106950 2024-03-26T01:56:54.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.827+0000 7f336affd700 1 --2- 172.21.15.72:0/2963326343 >> v2:172.21.15.72:6800/2923394687 conn(0x7f3354038f40 0x7f335403b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:54.828 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.827+0000 7f336affd700 1 --2- 172.21.15.72:0/2963326343 >> v2:172.21.15.72:6800/2923394687 conn(0x7f3354038f40 0x7f335403b400 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3360006fd0 tx=0x7f3360006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:54.831 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:54.832+0000 7f3368ff9700 1 -- 172.21.15.72:0/2963326343 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f335c015530 con 0x7f336c106950 2024-03-26T01:56:55.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:55.033+0000 7f3371d75700 1 -- 172.21.15.72:0/2963326343 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f336c103b20 con 0x7f336c106950 2024-03-26T01:56:55.084 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:55.084+0000 7f3368ff9700 1 -- 172.21.15.72:0/2963326343 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f335c026090 con 0x7f336c106950 2024-03-26T01:56:55.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:55.086+0000 7f3371d75700 1 -- 172.21.15.72:0/2963326343 >> v2:172.21.15.72:6800/2923394687 conn(0x7f3354038f40 msgr2=0x7f335403b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:55.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:55.086+0000 7f3371d75700 1 --2- 172.21.15.72:0/2963326343 >> v2:172.21.15.72:6800/2923394687 conn(0x7f3354038f40 0x7f335403b400 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f3360006fd0 tx=0x7f3360006e40 comp rx=0 tx=0).stop 2024-03-26T01:56:55.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:55.086+0000 7f3371d75700 1 -- 172.21.15.72:0/2963326343 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f336c106950 msgr2=0x7f336c102520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:55.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:55.086+0000 7f3371d75700 1 --2- 172.21.15.72:0/2963326343 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f336c106950 0x7f336c102520 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f335c00f690 tx=0x7f335c00f770 comp rx=0 tx=0).stop 2024-03-26T01:56:55.086 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:55.086+0000 7f3371d75700 1 -- 172.21.15.72:0/2963326343 shutdown_connections 2024-03-26T01:56:55.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:55.086+0000 7f3371d75700 1 --2- 172.21.15.72:0/2963326343 >> v2:172.21.15.72:6800/2923394687 conn(0x7f3354038f40 0x7f335403b400 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:55.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:55.086+0000 7f3371d75700 1 --2- 172.21.15.72:0/2963326343 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f336c106950 0x7f336c102520 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:55.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:55.086+0000 7f3371d75700 1 -- 172.21.15.72:0/2963326343 >> 172.21.15.72:0/2963326343 conn(0x7f336c0775c0 msgr2=0x7f336c108710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:55.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:55.086+0000 7f3371d75700 1 -- 172.21.15.72:0/2963326343 shutdown_connections 2024-03-26T01:56:55.087 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:56:55.086+0000 7f3371d75700 1 -- 172.21.15.72:0/2963326343 wait complete. 2024-03-26T01:56:55.087 INFO:teuthology.orchestra.run.smithi072.stderr:adjusted tunables profile to default 2024-03-26T01:56:55.417 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:55 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2963326343' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T01:56:55.758 INFO:tasks.cephadm:Adding mon.a on smithi072 2024-03-26T01:56:55.758 INFO:tasks.cephadm:Adding mon.c on smithi072 2024-03-26T01:56:55.758 INFO:tasks.cephadm:Adding mon.b on smithi173 2024-03-26T01:56:55.759 DEBUG:teuthology.orchestra.run.smithi173:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi072:172.21.15.72=a;smithi072:[v2:172.21.15.72:3301,v1:172.21.15.72:6790]=c;smithi173:172.21.15.173=b' 2024-03-26T01:56:55.994 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:56:56.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:56 smithi072 ceph-mon[30177]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:56.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:56 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2963326343' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T01:56:56.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:56 smithi072 ceph-mon[30177]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T01:56:56.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:56 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:56.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:56 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:56.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:56 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:56.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:56 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:56.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:56 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:56:56.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:56 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:56.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:56 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:56:57.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:57 smithi072 ceph-mon[30177]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-26T01:56:57.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:57 smithi072 ceph-mon[30177]: Updating smithi173:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:56:57.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:57 smithi072 ceph-mon[30177]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:56:57.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:57 smithi072 ceph-mon[30177]: Updating smithi173:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:56:57.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:57 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:57.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:57 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:57.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:57 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:57.518 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.516+0000 7f919eb6f700 1 -- 172.21.15.173:0/994975120 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9198100370 msgr2=0x7f9198100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:57.518 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.516+0000 7f919eb6f700 1 --2- 172.21.15.173:0/994975120 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9198100370 0x7f9198100750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9188009910 tx=0x7f9188009c20 comp rx=0 tx=0).stop 2024-03-26T01:56:57.518 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.517+0000 7f919eb6f700 1 -- 172.21.15.173:0/994975120 shutdown_connections 2024-03-26T01:56:57.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.517+0000 7f919eb6f700 1 --2- 172.21.15.173:0/994975120 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9198100370 0x7f9198100750 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:57.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.517+0000 7f919eb6f700 1 -- 172.21.15.173:0/994975120 >> 172.21.15.173:0/994975120 conn(0x7f91980fbc00 msgr2=0x7f91980fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:57.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.517+0000 7f919eb6f700 1 -- 172.21.15.173:0/994975120 shutdown_connections 2024-03-26T01:56:57.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.517+0000 7f919eb6f700 1 -- 172.21.15.173:0/994975120 wait complete. 2024-03-26T01:56:57.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.517+0000 7f919eb6f700 1 Processor -- start 2024-03-26T01:56:57.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.517+0000 7f919eb6f700 1 -- start start 2024-03-26T01:56:57.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.517+0000 7f919eb6f700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9198100370 0x7f9198197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:57.519 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.517+0000 7f919eb6f700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9198197e90 con 0x7f9198100370 2024-03-26T01:56:57.520 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.518+0000 7f919c90b700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9198100370 0x7f9198197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:57.520 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.518+0000 7f919c90b700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9198100370 0x7f9198197950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.173:41442/0 (socket says 172.21.15.173:41442) 2024-03-26T01:56:57.520 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.518+0000 7f919c90b700 1 -- 172.21.15.173:0/1218887308 learned_addr learned my addr 172.21.15.173:0/1218887308 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-26T01:56:57.522 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.518+0000 7f919c90b700 1 -- 172.21.15.173:0/1218887308 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91880095f0 con 0x7f9198100370 2024-03-26T01:56:57.522 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.518+0000 7f919c90b700 1 --2- 172.21.15.173:0/1218887308 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9198100370 0x7f9198197950 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f918800f690 tx=0x7f918800f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:57.522 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.518+0000 7f9195ffb700 1 -- 172.21.15.173:0/1218887308 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f918801d070 con 0x7f9198100370 2024-03-26T01:56:57.522 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.518+0000 7f919eb6f700 1 -- 172.21.15.173:0/1218887308 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9198198090 con 0x7f9198100370 2024-03-26T01:56:57.523 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.518+0000 7f9195ffb700 1 -- 172.21.15.173:0/1218887308 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f9188004500 con 0x7f9198100370 2024-03-26T01:56:57.523 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.518+0000 7f9195ffb700 1 -- 172.21.15.173:0/1218887308 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9188003b50 con 0x7f9198100370 2024-03-26T01:56:57.523 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.519+0000 7f919eb6f700 1 -- 172.21.15.173:0/1218887308 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91981984b0 con 0x7f9198100370 2024-03-26T01:56:57.523 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.519+0000 7f919eb6f700 1 -- 172.21.15.173:0/1218887308 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91980636b0 con 0x7f9198100370 2024-03-26T01:56:57.523 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.520+0000 7f9195ffb700 1 -- 172.21.15.173:0/1218887308 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) v1 ==== 45949+0+0 (secure 0 0 0) 0x7f9188004050 con 0x7f9198100370 2024-03-26T01:56:57.523 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.520+0000 7f9195ffb700 1 --2- 172.21.15.173:0/1218887308 >> v2:172.21.15.72:6800/2923394687 conn(0x7f9180038f40 0x7f918003b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:56:57.523 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.520+0000 7f9195ffb700 1 -- 172.21.15.173:0/1218887308 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (secure 0 0 0) 0x7f918804cd10 con 0x7f9198100370 2024-03-26T01:56:57.523 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.520+0000 7f9197fff700 1 --2- 172.21.15.173:0/1218887308 >> v2:172.21.15.72:6800/2923394687 conn(0x7f9180038f40 0x7f918003b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:56:57.523 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.520+0000 7f9197fff700 1 --2- 172.21.15.173:0/1218887308 >> v2:172.21.15.72:6800/2923394687 conn(0x7f9180038f40 0x7f918003b400 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f918c006fd0 tx=0x7f918c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:56:57.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.525+0000 7f9195ffb700 1 -- 172.21.15.173:0/1218887308 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f9188015530 con 0x7f9198100370 2024-03-26T01:56:57.729 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.729+0000 7f919eb6f700 1 -- 172.21.15.173:0/1218887308 --> v2:172.21.15.72:6800/2923394687 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi072:172.21.15.72=a;smithi072:[v2:172.21.15.72:3301,v1:172.21.15.72:6790]=c;smithi173:172.21.15.173=b", "target": ["mon-mgr", ""]}) v1 -- 0x7f91981986a0 con 0x7f9180038f40 2024-03-26T01:56:57.738 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.738+0000 7f9195ffb700 1 -- 172.21.15.173:0/1218887308 <== mgr.14152 v2:172.21.15.72:6800/2923394687 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f91981986a0 con 0x7f9180038f40 2024-03-26T01:56:57.738 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mon update... 2024-03-26T01:56:57.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.740+0000 7f919eb6f700 1 -- 172.21.15.173:0/1218887308 >> v2:172.21.15.72:6800/2923394687 conn(0x7f9180038f40 msgr2=0x7f918003b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:57.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.740+0000 7f919eb6f700 1 --2- 172.21.15.173:0/1218887308 >> v2:172.21.15.72:6800/2923394687 conn(0x7f9180038f40 0x7f918003b400 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f918c006fd0 tx=0x7f918c006e40 comp rx=0 tx=0).stop 2024-03-26T01:56:57.740 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.740+0000 7f919eb6f700 1 -- 172.21.15.173:0/1218887308 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9198100370 msgr2=0x7f9198197950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:56:57.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.740+0000 7f919eb6f700 1 --2- 172.21.15.173:0/1218887308 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9198100370 0x7f9198197950 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f918800f690 tx=0x7f918800f770 comp rx=0 tx=0).stop 2024-03-26T01:56:57.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.740+0000 7f919eb6f700 1 -- 172.21.15.173:0/1218887308 shutdown_connections 2024-03-26T01:56:57.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.740+0000 7f919eb6f700 1 --2- 172.21.15.173:0/1218887308 >> v2:172.21.15.72:6800/2923394687 conn(0x7f9180038f40 0x7f918003b400 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:57.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.740+0000 7f919eb6f700 1 --2- 172.21.15.173:0/1218887308 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9198100370 0x7f9198197950 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:56:57.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.740+0000 7f919eb6f700 1 -- 172.21.15.173:0/1218887308 >> 172.21.15.173:0/1218887308 conn(0x7f91980fbc00 msgr2=0x7f9198101d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:56:57.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.740+0000 7f919eb6f700 1 -- 172.21.15.173:0/1218887308 shutdown_connections 2024-03-26T01:56:57.741 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:56:57.740+0000 7f919eb6f700 1 -- 172.21.15.173:0/1218887308 wait complete. 2024-03-26T01:56:58.300 DEBUG:teuthology.orchestra.run.smithi072:mon.c> sudo journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@mon.c.service 2024-03-26T01:56:58.303 DEBUG:teuthology.orchestra.run.smithi173:mon.b> sudo journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@mon.b.service 2024-03-26T01:56:58.305 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T01:56:58.306 DEBUG:teuthology.orchestra.run.smithi173:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T01:56:58.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:58 smithi072 ceph-mon[30177]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:56:58.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:58 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:58.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:58 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:56:58.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:58 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:58.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:58 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:56:58.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:58 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:56:58.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:58 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:56:58.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:58 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:56:59.091 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:56:59.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:59 smithi072 ceph-mon[30177]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi072:172.21.15.72=a;smithi072:[v2:172.21.15.72:3301,v1:172.21.15.72:6790]=c;smithi173:172.21.15.173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:56:59.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:59 smithi072 ceph-mon[30177]: Saving service mon spec with placement smithi072:172.21.15.72=a;smithi072:[v2:172.21.15.72:3301,v1:172.21.15.72:6790]=c;smithi173:172.21.15.173=b;count:3 2024-03-26T01:56:59.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:56:59 smithi072 ceph-mon[30177]: Deploying daemon mon.b on smithi173 2024-03-26T01:57:00.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:00 smithi072 ceph-mon[30177]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:02.091 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.089+0000 7efcebfff700 1 -- 172.21.15.173:0/492740481 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efce4106940 msgr2=0x7efce4106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:02.091 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.089+0000 7efcebfff700 1 --2- 172.21.15.173:0/492740481 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efce4106940 0x7efce4106d20 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efcd4009910 tx=0x7efcd4009c20 comp rx=0 tx=0).stop 2024-03-26T01:57:02.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.089+0000 7efcebfff700 1 -- 172.21.15.173:0/492740481 shutdown_connections 2024-03-26T01:57:02.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.089+0000 7efcebfff700 1 --2- 172.21.15.173:0/492740481 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efce4106940 0x7efce4106d20 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:02.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.089+0000 7efcebfff700 1 -- 172.21.15.173:0/492740481 >> 172.21.15.173:0/492740481 conn(0x7efce40fc430 msgr2=0x7efce40fe850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:57:02.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.089+0000 7efcebfff700 1 -- 172.21.15.173:0/492740481 shutdown_connections 2024-03-26T01:57:02.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.089+0000 7efcebfff700 1 -- 172.21.15.173:0/492740481 wait complete. 2024-03-26T01:57:02.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.090+0000 7efcebfff700 1 Processor -- start 2024-03-26T01:57:02.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.090+0000 7efcebfff700 1 -- start start 2024-03-26T01:57:02.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.090+0000 7efcebfff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efce4106940 0x7efce419c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:57:02.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.090+0000 7efcebfff700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce419cc50 con 0x7efce4106940 2024-03-26T01:57:02.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.090+0000 7efce9d9b700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efce4106940 0x7efce419c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:57:02.092 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.090+0000 7efce9d9b700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efce4106940 0x7efce419c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.173:41458/0 (socket says 172.21.15.173:41458) 2024-03-26T01:57:02.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.090+0000 7efce9d9b700 1 -- 172.21.15.173:0/175388125 learned_addr learned my addr 172.21.15.173:0/175388125 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-26T01:57:02.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.090+0000 7efce9d9b700 1 -- 172.21.15.173:0/175388125 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd40095f0 con 0x7efce4106940 2024-03-26T01:57:02.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.091+0000 7efce9d9b700 1 --2- 172.21.15.173:0/175388125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efce4106940 0x7efce419c710 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efcd400f690 tx=0x7efcd400f770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:57:02.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.091+0000 7efcdaffd700 1 -- 172.21.15.173:0/175388125 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efcd401d070 con 0x7efce4106940 2024-03-26T01:57:02.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.091+0000 7efcebfff700 1 -- 172.21.15.173:0/175388125 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efce419ceb0 con 0x7efce4106940 2024-03-26T01:57:02.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.091+0000 7efcdaffd700 1 -- 172.21.15.173:0/175388125 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7efcd4004500 con 0x7efce4106940 2024-03-26T01:57:02.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.091+0000 7efcdaffd700 1 -- 172.21.15.173:0/175388125 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efcd4003b50 con 0x7efce4106940 2024-03-26T01:57:02.093 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.091+0000 7efcebfff700 1 -- 172.21.15.173:0/175388125 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efce419d330 con 0x7efce4106940 2024-03-26T01:57:02.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.092+0000 7efcebfff700 1 -- 172.21.15.173:0/175388125 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efce40636b0 con 0x7efce4106940 2024-03-26T01:57:02.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.092+0000 7efcdaffd700 1 -- 172.21.15.173:0/175388125 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) v1 ==== 45949+0+0 (secure 0 0 0) 0x7efcd4004050 con 0x7efce4106940 2024-03-26T01:57:02.094 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.092+0000 7efcdaffd700 1 --2- 172.21.15.173:0/175388125 >> v2:172.21.15.72:6800/2923394687 conn(0x7efcd0038f40 0x7efcd003b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:57:02.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.092+0000 7efcdaffd700 1 -- 172.21.15.173:0/175388125 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (secure 0 0 0) 0x7efcd404ccc0 con 0x7efce4106940 2024-03-26T01:57:02.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.092+0000 7efce959a700 1 --2- 172.21.15.173:0/175388125 >> v2:172.21.15.72:6800/2923394687 conn(0x7efcd0038f40 0x7efcd003b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:57:02.095 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.093+0000 7efce959a700 1 --2- 172.21.15.173:0/175388125 >> v2:172.21.15.72:6800/2923394687 conn(0x7efcd0038f40 0x7efcd003b400 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efce0006fd0 tx=0x7efce0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:57:02.097 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.097+0000 7efcdaffd700 1 -- 172.21.15.173:0/175388125 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7efcd4015530 con 0x7efce4106940 2024-03-26T01:57:02.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.397+0000 7efcebfff700 1 -- 172.21.15.173:0/175388125 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efce410a450 con 0x7efce4106940 2024-03-26T01:57:02.397 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.397+0000 7efcdaffd700 1 -- 172.21.15.173:0/175388125 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+745 (secure 0 0 0) 0x7efcd4026020 con 0x7efce4106940 2024-03-26T01:57:02.398 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:57:02.398 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"c86fbdd4-eb13-11ee-95d0-87774f69a715","modified":"2024-03-26T01:55:12.142924Z","created":"2024-03-26T01:55:12.142924Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:3300","nonce":0},{"type":"v1","addr":"172.21.15.72:6789","nonce":0}]},"addr":"172.21.15.72:6789/0","public_addr":"172.21.15.72:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T01:57:02.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.399+0000 7efcebfff700 1 -- 172.21.15.173:0/175388125 >> v2:172.21.15.72:6800/2923394687 conn(0x7efcd0038f40 msgr2=0x7efcd003b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:02.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.399+0000 7efcebfff700 1 --2- 172.21.15.173:0/175388125 >> v2:172.21.15.72:6800/2923394687 conn(0x7efcd0038f40 0x7efcd003b400 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7efce0006fd0 tx=0x7efce0006e40 comp rx=0 tx=0).stop 2024-03-26T01:57:02.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.399+0000 7efcebfff700 1 -- 172.21.15.173:0/175388125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efce4106940 msgr2=0x7efce419c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:02.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.399+0000 7efcebfff700 1 --2- 172.21.15.173:0/175388125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efce4106940 0x7efce419c710 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efcd400f690 tx=0x7efcd400f770 comp rx=0 tx=0).stop 2024-03-26T01:57:02.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.400+0000 7efcebfff700 1 -- 172.21.15.173:0/175388125 shutdown_connections 2024-03-26T01:57:02.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.400+0000 7efcebfff700 1 --2- 172.21.15.173:0/175388125 >> v2:172.21.15.72:6800/2923394687 conn(0x7efcd0038f40 0x7efcd003b400 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:02.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.400+0000 7efcebfff700 1 --2- 172.21.15.173:0/175388125 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7efce4106940 0x7efce419c710 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:02.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.400+0000 7efcebfff700 1 -- 172.21.15.173:0/175388125 >> 172.21.15.173:0/175388125 conn(0x7efce40fc430 msgr2=0x7efce41086e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:57:02.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.400+0000 7efcebfff700 1 -- 172.21.15.173:0/175388125 shutdown_connections 2024-03-26T01:57:02.400 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:02.400+0000 7efcebfff700 1 -- 172.21.15.173:0/175388125 wait complete. 2024-03-26T01:57:02.401 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-03-26T01:57:02.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:02 smithi072 ceph-mon[30177]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:03.296 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 podman[33802]: 2024-03-26 01:57:03.19697654 +0000 UTC m=+0.124951487 container create 5d6891919918325caca9b0924f43b96649732458d28e4245bf2f555615ba61e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-26T01:57:03.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.173:0/175388125' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T01:57:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 podman[33802]: 2024-03-26 01:57:03.374335362 +0000 UTC m=+0.302310308 container init 5d6891919918325caca9b0924f43b96649732458d28e4245bf2f555615ba61e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-b, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-26T01:57:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 podman[33802]: 2024-03-26 01:57:03.380215397 +0000 UTC m=+0.308190342 container start 5d6891919918325caca9b0924f43b96649732458d28e4245bf2f555615ba61e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T01:57:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T01:57:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 6 2024-03-26T01:57:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: pidfile_write: ignore empty --pid-file 2024-03-26T01:57:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: load: jerasure load: lrc 2024-03-26T01:57:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: RocksDB version: 7.9.2 2024-03-26T01:57:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Git sha 0 2024-03-26T01:57:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T01:57:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: DB SUMMARY 2024-03-26T01:57:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: DB Session ID: 3JX0NCJV4FTYIFZI0JEF 2024-03-26T01:57:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: CURRENT file: CURRENT 2024-03-26T01:57:03.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T01:57:03.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-26T01:57:03.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-26T01:57:03.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-03-26T01:57:03.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.error_if_exists: 0 2024-03-26T01:57:03.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.create_if_missing: 0 2024-03-26T01:57:03.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.paranoid_checks: 1 2024-03-26T01:57:03.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T01:57:03.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T01:57:03.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T01:57:03.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.env: 0x55a2efef7ae0 2024-03-26T01:57:03.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T01:57:03.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.info_log: 0x55a2f1811920 2024-03-26T01:57:03.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T01:57:03.816 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.statistics: (nil) 2024-03-26T01:57:03.816 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.use_fsync: 0 2024-03-26T01:57:03.816 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.max_log_file_size: 0 2024-03-26T01:57:03.816 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T01:57:03.816 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T01:57:03.816 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T01:57:03.816 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T01:57:03.816 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.allow_fallocate: 1 2024-03-26T01:57:03.817 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T01:57:03.817 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T01:57:03.817 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.use_direct_reads: 0 2024-03-26T01:57:03.817 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T01:57:03.817 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T01:57:03.817 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.db_log_dir: 2024-03-26T01:57:03.817 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.wal_dir: 2024-03-26T01:57:03.817 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T01:57:03.817 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T01:57:03.817 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T01:57:03.817 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T01:57:03.817 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T01:57:03.817 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T01:57:03.817 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T01:57:03.818 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T01:57:03.818 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.write_buffer_manager: 0x55a2f17e41e0 2024-03-26T01:57:03.818 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T01:57:03.818 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T01:57:03.818 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T01:57:03.818 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T01:57:03.818 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T01:57:03.818 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T01:57:03.818 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T01:57:03.818 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T01:57:03.818 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.unordered_write: 0 2024-03-26T01:57:03.818 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T01:57:03.818 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T01:57:03.818 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T01:57:03.819 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T01:57:03.819 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.row_cache: None 2024-03-26T01:57:03.819 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.wal_filter: None 2024-03-26T01:57:03.819 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T01:57:03.819 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T01:57:03.819 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.two_write_queues: 0 2024-03-26T01:57:03.819 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T01:57:03.819 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.wal_compression: 0 2024-03-26T01:57:03.819 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.atomic_flush: 0 2024-03-26T01:57:03.819 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T01:57:03.819 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T01:57:03.819 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T01:57:03.819 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.log_readahead_size: 0 2024-03-26T01:57:03.819 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T01:57:03.819 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T01:57:03.819 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T01:57:03.820 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T01:57:03.820 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T01:57:03.820 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T01:57:03.820 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T01:57:03.820 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.max_background_jobs: 2 2024-03-26T01:57:03.820 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.max_background_compactions: -1 2024-03-26T01:57:03.820 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.max_subcompactions: 1 2024-03-26T01:57:03.820 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T01:57:03.820 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T01:57:03.820 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T01:57:03.820 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T01:57:03.820 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T01:57:03.820 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T01:57:03.820 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T01:57:03.821 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T01:57:03.821 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.max_open_files: -1 2024-03-26T01:57:03.821 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T01:57:03.821 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T01:57:03.821 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T01:57:03.821 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T01:57:03.821 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.max_background_flushes: -1 2024-03-26T01:57:03.821 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Compression algorithms supported: 2024-03-26T01:57:03.821 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T01:57:03.821 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: kZSTD supported: 0 2024-03-26T01:57:03.821 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: kXpressCompression supported: 0 2024-03-26T01:57:03.821 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T01:57:03.821 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: kZlibCompression supported: 1 2024-03-26T01:57:03.821 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: kSnappyCompression supported: 1 2024-03-26T01:57:03.822 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: kLZ4Compression supported: 1 2024-03-26T01:57:03.822 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: kBZip2Compression supported: 0 2024-03-26T01:57:03.822 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T01:57:03.822 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T01:57:03.822 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-26T01:57:03.822 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T01:57:03.822 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T01:57:03.822 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.merge_operator: 2024-03-26T01:57:03.822 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.compaction_filter: None 2024-03-26T01:57:03.822 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T01:57:03.822 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T01:57:03.822 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T01:57:03.822 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T01:57:03.822 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55a2f1811a20) 2024-03-26T01:57:03.823 INFO:journalctl@ceph.mon.b.smithi173.stdout: cache_index_and_filter_blocks: 1 2024-03-26T01:57:03.823 INFO:journalctl@ceph.mon.b.smithi173.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T01:57:03.823 INFO:journalctl@ceph.mon.b.smithi173.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T01:57:03.823 INFO:journalctl@ceph.mon.b.smithi173.stdout: pin_top_level_index_and_filter: 1 2024-03-26T01:57:03.823 INFO:journalctl@ceph.mon.b.smithi173.stdout: index_type: 0 2024-03-26T01:57:03.823 INFO:journalctl@ceph.mon.b.smithi173.stdout: data_block_index_type: 0 2024-03-26T01:57:03.823 INFO:journalctl@ceph.mon.b.smithi173.stdout: index_shortening: 1 2024-03-26T01:57:03.823 INFO:journalctl@ceph.mon.b.smithi173.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T01:57:03.823 INFO:journalctl@ceph.mon.b.smithi173.stdout: checksum: 4 2024-03-26T01:57:03.823 INFO:journalctl@ceph.mon.b.smithi173.stdout: no_block_cache: 0 2024-03-26T01:57:03.823 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_cache: 0x55a2f1815090 2024-03-26T01:57:03.823 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_cache_name: BinnedLRUCache 2024-03-26T01:57:03.823 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_cache_options: 2024-03-26T01:57:03.823 INFO:journalctl@ceph.mon.b.smithi173.stdout: capacity : 536870912 2024-03-26T01:57:03.823 INFO:journalctl@ceph.mon.b.smithi173.stdout: num_shard_bits : 4 2024-03-26T01:57:03.823 INFO:journalctl@ceph.mon.b.smithi173.stdout: strict_capacity_limit : 0 2024-03-26T01:57:03.824 INFO:journalctl@ceph.mon.b.smithi173.stdout: high_pri_pool_ratio: 0.000 2024-03-26T01:57:03.824 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_cache_compressed: (nil) 2024-03-26T01:57:03.824 INFO:journalctl@ceph.mon.b.smithi173.stdout: persistent_cache: (nil) 2024-03-26T01:57:03.824 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_size: 4096 2024-03-26T01:57:03.824 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_size_deviation: 10 2024-03-26T01:57:03.824 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_restart_interval: 16 2024-03-26T01:57:03.824 INFO:journalctl@ceph.mon.b.smithi173.stdout: index_block_restart_interval: 1 2024-03-26T01:57:03.824 INFO:journalctl@ceph.mon.b.smithi173.stdout: metadata_block_size: 4096 2024-03-26T01:57:03.824 INFO:journalctl@ceph.mon.b.smithi173.stdout: partition_filters: 0 2024-03-26T01:57:03.824 INFO:journalctl@ceph.mon.b.smithi173.stdout: use_delta_encoding: 1 2024-03-26T01:57:03.824 INFO:journalctl@ceph.mon.b.smithi173.stdout: filter_policy: bloomfilter 2024-03-26T01:57:03.824 INFO:journalctl@ceph.mon.b.smithi173.stdout: whole_key_filtering: 1 2024-03-26T01:57:03.824 INFO:journalctl@ceph.mon.b.smithi173.stdout: verify_compression: 0 2024-03-26T01:57:03.824 INFO:journalctl@ceph.mon.b.smithi173.stdout: read_amp_bytes_per_bit: 0 2024-03-26T01:57:03.824 INFO:journalctl@ceph.mon.b.smithi173.stdout: format_version: 5 2024-03-26T01:57:03.825 INFO:journalctl@ceph.mon.b.smithi173.stdout: enable_index_compression: 1 2024-03-26T01:57:03.825 INFO:journalctl@ceph.mon.b.smithi173.stdout: block_align: 0 2024-03-26T01:57:03.825 INFO:journalctl@ceph.mon.b.smithi173.stdout: max_auto_readahead_size: 262144 2024-03-26T01:57:03.825 INFO:journalctl@ceph.mon.b.smithi173.stdout: prepopulate_block_cache: 0 2024-03-26T01:57:03.825 INFO:journalctl@ceph.mon.b.smithi173.stdout: initial_auto_readahead_size: 8192 2024-03-26T01:57:03.825 INFO:journalctl@ceph.mon.b.smithi173.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T01:57:03.825 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T01:57:03.825 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T01:57:03.825 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.compression: NoCompression 2024-03-26T01:57:03.825 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T01:57:03.825 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T01:57:03.825 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T01:57:03.825 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.num_levels: 7 2024-03-26T01:57:03.825 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T01:57:03.825 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T01:57:03.826 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T01:57:03.826 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T01:57:03.826 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T01:57:03.826 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T01:57:03.826 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T01:57:03.826 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T01:57:03.826 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T01:57:03.826 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T01:57:03.826 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T01:57:03.826 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T01:57:03.826 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T01:57:03.826 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T01:57:03.826 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T01:57:03.826 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T01:57:03.826 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T01:57:03.826 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T01:57:03.827 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T01:57:03.827 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T01:57:03.827 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T01:57:03.827 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T01:57:03.827 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T01:57:03.827 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T01:57:03.827 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T01:57:03.827 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T01:57:03.827 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T01:57:03.827 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T01:57:03.827 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T01:57:03.827 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T01:57:03.827 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T01:57:03.827 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T01:57:03.827 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T01:57:03.828 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T01:57:03.828 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T01:57:03.828 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T01:57:03.828 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T01:57:03.828 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T01:57:03.828 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T01:57:03.828 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T01:57:03.828 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T01:57:03.828 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T01:57:03.828 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T01:57:03.828 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T01:57:03.828 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T01:57:03.828 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T01:57:03.828 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T01:57:03.828 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T01:57:03.828 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T01:57:03.829 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T01:57:03.829 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T01:57:03.829 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T01:57:03.829 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T01:57:03.829 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.table_properties_collectors: 2024-03-26T01:57:03.829 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.inplace_update_support: 0 2024-03-26T01:57:03.829 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T01:57:03.829 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T01:57:03.829 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T01:57:03.829 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T01:57:03.829 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.bloom_locality: 0 2024-03-26T01:57:03.829 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.max_successive_merges: 0 2024-03-26T01:57:03.829 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T01:57:03.830 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T01:57:03.830 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T01:57:03.830 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T01:57:03.830 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.ttl: 2592000 2024-03-26T01:57:03.830 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T01:57:03.830 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T01:57:03.830 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T01:57:03.830 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.enable_blob_files: false 2024-03-26T01:57:03.830 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.min_blob_size: 0 2024-03-26T01:57:03.830 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T01:57:03.830 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T01:57:03.830 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T01:57:03.830 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T01:57:03.830 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T01:57:03.830 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T01:57:03.831 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T01:57:03.831 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T01:57:03.831 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T01:57:03.831 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-26T01:57:03.831 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 177895a2-9adb-4c63-9c6d-86bdbc6d3b82 2024-03-26T01:57:03.831 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418223410655, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-26T01:57:03.831 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-26T01:57:03.831 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418223411167, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711418223, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "177895a2-9adb-4c63-9c6d-86bdbc6d3b82", "db_session_id": "3JX0NCJV4FTYIFZI0JEF", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-26T01:57:03.831 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418223411266, "job": 1, "event": "recovery_finished"} 2024-03-26T01:57:03.831 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-26T01:57:03.831 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T01:57:03.831 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55a2f1906000 2024-03-26T01:57:03.831 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: DB pointer 0x55a2f18f0000 2024-03-26T01:57:03.831 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T01:57:03.832 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T01:57:03.832 INFO:journalctl@ceph.mon.b.smithi173.stdout: ** DB Stats ** 2024-03-26T01:57:03.832 INFO:journalctl@ceph.mon.b.smithi173.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T01:57:03.832 INFO:journalctl@ceph.mon.b.smithi173.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T01:57:03.832 INFO:journalctl@ceph.mon.b.smithi173.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T01:57:03.832 INFO:journalctl@ceph.mon.b.smithi173.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T01:57:03.832 INFO:journalctl@ceph.mon.b.smithi173.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T01:57:03.832 INFO:journalctl@ceph.mon.b.smithi173.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T01:57:03.832 INFO:journalctl@ceph.mon.b.smithi173.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T01:57:03.832 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-03-26T01:57:03.832 INFO:journalctl@ceph.mon.b.smithi173.stdout: ** Compaction Stats [default] ** 2024-03-26T01:57:03.832 INFO:journalctl@ceph.mon.b.smithi173.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T01:57:03.832 INFO:journalctl@ceph.mon.b.smithi173.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T01:57:03.833 INFO:journalctl@ceph.mon.b.smithi173.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T01:57:03.833 INFO:journalctl@ceph.mon.b.smithi173.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T01:57:03.833 INFO:journalctl@ceph.mon.b.smithi173.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T01:57:03.833 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-03-26T01:57:03.833 INFO:journalctl@ceph.mon.b.smithi173.stdout: ** Compaction Stats [default] ** 2024-03-26T01:57:03.833 INFO:journalctl@ceph.mon.b.smithi173.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T01:57:03.833 INFO:journalctl@ceph.mon.b.smithi173.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T01:57:03.833 INFO:journalctl@ceph.mon.b.smithi173.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T01:57:03.833 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-03-26T01:57:03.833 INFO:journalctl@ceph.mon.b.smithi173.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T01:57:03.833 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-03-26T01:57:03.833 INFO:journalctl@ceph.mon.b.smithi173.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T01:57:03.833 INFO:journalctl@ceph.mon.b.smithi173.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T01:57:03.833 INFO:journalctl@ceph.mon.b.smithi173.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T01:57:03.833 INFO:journalctl@ceph.mon.b.smithi173.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T01:57:03.834 INFO:journalctl@ceph.mon.b.smithi173.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T01:57:03.834 INFO:journalctl@ceph.mon.b.smithi173.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T01:57:03.834 INFO:journalctl@ceph.mon.b.smithi173.stdout: Cumulative compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T01:57:03.834 INFO:journalctl@ceph.mon.b.smithi173.stdout: Interval compaction: 0.00 GB write, 0.51 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T01:57:03.834 INFO:journalctl@ceph.mon.b.smithi173.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T01:57:03.834 INFO:journalctl@ceph.mon.b.smithi173.stdout: Block cache BinnedLRUCache@0x55a2f1815090#6 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-26T01:57:03.834 INFO:journalctl@ceph.mon.b.smithi173.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-26T01:57:03.834 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-03-26T01:57:03.834 INFO:journalctl@ceph.mon.b.smithi173.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T01:57:03.834 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-26T01:57:03.834 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: using public_addr v2:172.21.15.173:0/0 -> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-03-26T01:57:03.834 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: starting mon.b rank -1 at public addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] at bind addrs [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T01:57:03.834 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: mon.b@-1(???) e0 preinit fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T01:57:03.835 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: mon.b@-1(synchronizing).mds e1 new map 2024-03-26T01:57:03.835 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-26T01:57:03.835 INFO:journalctl@ceph.mon.b.smithi173.stdout: e1 2024-03-26T01:57:03.835 INFO:journalctl@ceph.mon.b.smithi173.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T01:57:03.835 INFO:journalctl@ceph.mon.b.smithi173.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-26T01:57:03.835 INFO:journalctl@ceph.mon.b.smithi173.stdout: legacy client fscid: -1 2024-03-26T01:57:03.835 INFO:journalctl@ceph.mon.b.smithi173.stdout: 2024-03-26T01:57:03.835 INFO:journalctl@ceph.mon.b.smithi173.stdout: No filesystems configured 2024-03-26T01:57:03.835 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-26T01:57:03.835 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-26T01:57:03.835 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-26T01:57:03.835 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-26T01:57:03.836 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-26T01:57:03.836 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-26T01:57:03.836 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T01:57:03.836 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T01:57:03.836 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T01:57:03.836 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T01:57:03.836 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2963326343' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T01:57:03.836 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:03.836 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2963326343' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T01:57:03.836 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T01:57:03.836 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:03.836 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:03.836 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:03.836 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:03.836 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:57:03.837 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:03.837 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:03.837 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-26T01:57:03.837 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: Updating smithi173:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:57:03.837 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:57:03.837 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: Updating smithi173:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:57:03.837 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:03.837 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:03.837 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:03.837 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:03.837 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:03.837 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:03.837 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:03.837 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:03.837 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:03.838 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:57:03.838 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:03.838 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi072:172.21.15.72=a;smithi072:[v2:172.21.15.72:3301,v1:172.21.15.72:6790]=c;smithi173:172.21.15.173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:03.838 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: Saving service mon spec with placement smithi072:172.21.15.72=a;smithi072:[v2:172.21.15.72:3301,v1:172.21.15.72:6790]=c;smithi173:172.21.15.173=b;count:3 2024-03-26T01:57:03.838 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: Deploying daemon mon.b on smithi173 2024-03-26T01:57:03.838 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:03.838 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:03.838 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.173:0/175388125' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T01:57:03.838 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 ceph-mon[33839]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-26T01:57:03.838 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 bash[33802]: 5d6891919918325caca9b0924f43b96649732458d28e4245bf2f555615ba61e7 2024-03-26T01:57:03.838 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:03 smithi173 systemd[1]: Started Ceph mon.b for c86fbdd4-eb13-11ee-95d0-87774f69a715. 2024-03-26T01:57:04.014 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T01:57:04.015 DEBUG:teuthology.orchestra.run.smithi173:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T01:57:04.252 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T01:57:08.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.796+0000 7fdb73ee3700 1 -- 172.21.15.173:0/568191550 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb580035c0 msgr2=0x7fdb58005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:08.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.796+0000 7fdb73ee3700 1 --2- 172.21.15.173:0/568191550 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb580035c0 0x7fdb58005a50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdb6c100970 tx=0x7fdb5c00feb0 comp rx=0 tx=0).stop 2024-03-26T01:57:08.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.796+0000 7fdb73ee3700 1 -- 172.21.15.173:0/568191550 shutdown_connections 2024-03-26T01:57:08.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.796+0000 7fdb73ee3700 1 --2- 172.21.15.173:0/568191550 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb580035c0 0x7fdb58005a50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:08.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.796+0000 7fdb73ee3700 1 -- 172.21.15.173:0/568191550 >> 172.21.15.173:0/568191550 conn(0x7fdb6c0fc530 msgr2=0x7fdb6c0fe950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:57:08.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.796+0000 7fdb73ee3700 1 -- 172.21.15.173:0/568191550 shutdown_connections 2024-03-26T01:57:08.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.796+0000 7fdb73ee3700 1 -- 172.21.15.173:0/568191550 wait complete. 2024-03-26T01:57:08.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.797+0000 7fdb73ee3700 1 Processor -- start 2024-03-26T01:57:08.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.797+0000 7fdb73ee3700 1 -- start start 2024-03-26T01:57:08.799 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:08 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:57:08.799 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:08 smithi173 ceph-mon[33839]: mon.a calling monitor election 2024-03-26T01:57:08.799 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:08 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:57:08.799 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:08 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:57:08.799 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:08 smithi173 ceph-mon[33839]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:08.799 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:08 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:57:08.799 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:08 smithi173 ceph-mon[33839]: mon.b calling monitor election 2024-03-26T01:57:08.799 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:08 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:57:08.799 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:08 smithi173 ceph-mon[33839]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:08.799 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:08 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:57:08.799 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:08 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:57:08.799 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:08 smithi173 ceph-mon[33839]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-26T01:57:08.799 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:08 smithi173 ceph-mon[33839]: monmap e2: 2 mons at {a=[v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T01:57:08.799 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:08 smithi173 ceph-mon[33839]: fsmap 2024-03-26T01:57:08.800 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:08 smithi173 ceph-mon[33839]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T01:57:08.800 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:08 smithi173 ceph-mon[33839]: mgrmap e13: y(active, since 41s) 2024-03-26T01:57:08.800 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:08 smithi173 ceph-mon[33839]: overall HEALTH_OK 2024-03-26T01:57:08.800 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:08 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:08.800 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:08 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:08.800 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:08 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:08.800 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:08 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:57:08.800 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:08 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:08.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.797+0000 7fdb73ee3700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb580035c0 0x7fdb6c195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:57:08.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.797+0000 7fdb73ee3700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb6c195e60 0x7fdb6c1962e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:57:08.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.797+0000 7fdb73ee3700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6c06a780 con 0x7fdb580035c0 2024-03-26T01:57:08.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.797+0000 7fdb73ee3700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6c06a8f0 con 0x7fdb6c195e60 2024-03-26T01:57:08.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.797+0000 7fdb71c7f700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb580035c0 0x7fdb6c195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:57:08.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.797+0000 7fdb7147e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb6c195e60 0x7fdb6c1962e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:57:08.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.797+0000 7fdb71c7f700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb580035c0 0x7fdb6c195920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.173:38026/0 (socket says 172.21.15.173:38026) 2024-03-26T01:57:08.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.798+0000 7fdb71c7f700 1 -- 172.21.15.173:0/265850551 learned_addr learned my addr 172.21.15.173:0/265850551 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-26T01:57:08.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.798+0000 7fdb7147e700 1 -- 172.21.15.173:0/265850551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb6c195e60 msgr2=0x7fdb6c1962e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-26T01:57:08.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.798+0000 7fdb7147e700 1 -- 172.21.15.173:0/265850551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb6c195e60 msgr2=0x7fdb6c1962e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-26T01:57:08.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.798+0000 7fdb7147e700 1 --2- 172.21.15.173:0/265850551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb6c195e60 0x7fdb6c1962e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-26T01:57:08.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.798+0000 7fdb7147e700 1 --2- 172.21.15.173:0/265850551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb6c195e60 0x7fdb6c1962e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T01:57:08.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.798+0000 7fdb71c7f700 1 -- 172.21.15.173:0/265850551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb6c195e60 msgr2=0x7fdb6c1962e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T01:57:08.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.798+0000 7fdb71c7f700 1 --2- 172.21.15.173:0/265850551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb6c195e60 0x7fdb6c1962e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:08.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.798+0000 7fdb71c7f700 1 -- 172.21.15.173:0/265850551 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5c00f900 con 0x7fdb580035c0 2024-03-26T01:57:08.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.798+0000 7fdb71c7f700 1 --2- 172.21.15.173:0/265850551 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb580035c0 0x7fdb6c195920 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c011490 tx=0x7fdb5c004f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:57:08.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.798+0000 7fdb62ffd700 1 -- 172.21.15.173:0/265850551 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fdb5c013960 con 0x7fdb580035c0 2024-03-26T01:57:08.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.798+0000 7fdb62ffd700 1 -- 172.21.15.173:0/265850551 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fdb5c004030 con 0x7fdb580035c0 2024-03-26T01:57:08.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.798+0000 7fdb73ee3700 1 -- 172.21.15.173:0/265850551 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb6c1a2f90 con 0x7fdb580035c0 2024-03-26T01:57:08.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.798+0000 7fdb62ffd700 1 -- 172.21.15.173:0/265850551 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fdb5c003ba0 con 0x7fdb580035c0 2024-03-26T01:57:08.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.799+0000 7fdb73ee3700 1 -- 172.21.15.173:0/265850551 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb6c1a33a0 con 0x7fdb580035c0 2024-03-26T01:57:08.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.799+0000 7fdb62ffd700 1 -- 172.21.15.173:0/265850551 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) v1 ==== 45949+0+0 (secure 0 0 0) 0x7fdb5c011510 con 0x7fdb580035c0 2024-03-26T01:57:08.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.800+0000 7fdb62ffd700 1 --2- 172.21.15.173:0/265850551 >> v2:172.21.15.72:6800/2923394687 conn(0x7fdb50038ee0 0x7fdb5003b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:57:08.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.800+0000 7fdb62ffd700 1 -- 172.21.15.173:0/265850551 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (secure 0 0 0) 0x7fdb5c04f3b0 con 0x7fdb580035c0 2024-03-26T01:57:08.803 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.800+0000 7fdb7147e700 1 --2- 172.21.15.173:0/265850551 >> v2:172.21.15.72:6800/2923394687 conn(0x7fdb50038ee0 0x7fdb5003b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:57:08.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.800+0000 7fdb73ee3700 1 -- 172.21.15.173:0/265850551 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb6c18f8d0 con 0x7fdb580035c0 2024-03-26T01:57:08.804 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.801+0000 7fdb7147e700 1 --2- 172.21.15.173:0/265850551 >> v2:172.21.15.72:6800/2923394687 conn(0x7fdb50038ee0 0x7fdb5003b3a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdb68009850 tx=0x7fdb68008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:57:08.810 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:08.810+0000 7fdb62ffd700 1 -- 172.21.15.173:0/265850551 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fdb5c0109c0 con 0x7fdb580035c0 2024-03-26T01:57:08.874 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:08 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:57:08.875 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:08 smithi072 ceph-mon[30177]: mon.a calling monitor election 2024-03-26T01:57:08.875 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:08 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:57:08.875 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:08 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:57:08.875 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:08 smithi072 ceph-mon[30177]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:08.875 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:08 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:57:08.875 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:08 smithi072 ceph-mon[30177]: mon.b calling monitor election 2024-03-26T01:57:08.875 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:08 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:57:08.875 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:08 smithi072 ceph-mon[30177]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:08.875 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:08 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:57:08.875 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:08 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:57:08.875 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:08 smithi072 ceph-mon[30177]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-26T01:57:08.875 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:08 smithi072 ceph-mon[30177]: monmap e2: 2 mons at {a=[v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T01:57:08.875 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:08 smithi072 ceph-mon[30177]: fsmap 2024-03-26T01:57:08.875 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:08 smithi072 ceph-mon[30177]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T01:57:08.876 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:08 smithi072 ceph-mon[30177]: mgrmap e13: y(active, since 41s) 2024-03-26T01:57:08.876 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:08 smithi072 ceph-mon[30177]: overall HEALTH_OK 2024-03-26T01:57:08.876 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:08 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:08.876 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:08 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:08.876 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:08 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:08.876 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:08 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:57:08.876 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:08 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:09.087 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:09.087+0000 7fdb73ee3700 1 -- 172.21.15.173:0/265850551 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdb6c06a010 con 0x7fdb580035c0 2024-03-26T01:57:09.088 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:09.087+0000 7fdb62ffd700 1 -- 172.21.15.173:0/265850551 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1015 (secure 0 0 0) 0x7fdb5c0103d0 con 0x7fdb580035c0 2024-03-26T01:57:09.088 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:57:09.088 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":2,"fsid":"c86fbdd4-eb13-11ee-95d0-87774f69a715","modified":"2024-03-26T01:57:03.433488Z","created":"2024-03-26T01:55:12.142924Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:3300","nonce":0},{"type":"v1","addr":"172.21.15.72:6789","nonce":0}]},"addr":"172.21.15.72:6789/0","public_addr":"172.21.15.72:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T01:57:09.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:09.089+0000 7fdb73ee3700 1 -- 172.21.15.173:0/265850551 >> v2:172.21.15.72:6800/2923394687 conn(0x7fdb50038ee0 msgr2=0x7fdb5003b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:09.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:09.089+0000 7fdb73ee3700 1 --2- 172.21.15.173:0/265850551 >> v2:172.21.15.72:6800/2923394687 conn(0x7fdb50038ee0 0x7fdb5003b3a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdb68009850 tx=0x7fdb68008040 comp rx=0 tx=0).stop 2024-03-26T01:57:09.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:09.090+0000 7fdb73ee3700 1 -- 172.21.15.173:0/265850551 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb580035c0 msgr2=0x7fdb6c195920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:09.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:09.090+0000 7fdb73ee3700 1 --2- 172.21.15.173:0/265850551 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb580035c0 0x7fdb6c195920 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c011490 tx=0x7fdb5c004f20 comp rx=0 tx=0).stop 2024-03-26T01:57:09.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:09.090+0000 7fdb73ee3700 1 -- 172.21.15.173:0/265850551 shutdown_connections 2024-03-26T01:57:09.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:09.090+0000 7fdb73ee3700 1 --2- 172.21.15.173:0/265850551 >> v2:172.21.15.72:6800/2923394687 conn(0x7fdb50038ee0 0x7fdb5003b3a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:09.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:09.090+0000 7fdb73ee3700 1 --2- 172.21.15.173:0/265850551 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdb580035c0 0x7fdb6c195920 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:09.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:09.090+0000 7fdb73ee3700 1 --2- 172.21.15.173:0/265850551 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb6c195e60 0x7fdb6c1962e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:09.090 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:09.090+0000 7fdb73ee3700 1 -- 172.21.15.173:0/265850551 >> 172.21.15.173:0/265850551 conn(0x7fdb6c0fc530 msgr2=0x7fdb6c0fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:57:09.091 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:09.090+0000 7fdb73ee3700 1 -- 172.21.15.173:0/265850551 shutdown_connections 2024-03-26T01:57:09.091 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:09.090+0000 7fdb73ee3700 1 -- 172.21.15.173:0/265850551 wait complete. 2024-03-26T01:57:09.091 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 2 2024-03-26T01:57:09.654 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:09 smithi173 ceph-mon[33839]: Deploying daemon mon.c on smithi072 2024-03-26T01:57:09.654 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:09 smithi173 ceph-mon[33839]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:09.655 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.173:0/265850551' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T01:57:09.655 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:09 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:57:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:09 smithi072 ceph-mon[30177]: Deploying daemon mon.c on smithi072 2024-03-26T01:57:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:09 smithi072 ceph-mon[30177]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.173:0/265850551' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T01:57:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:09 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:57:10.657 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T01:57:10.657 DEBUG:teuthology.orchestra.run.smithi173:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T01:57:10.903 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T01:57:11.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:11 smithi072 systemd[1]: Starting Ceph mon.c for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T01:57:12.008 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:11 smithi072 podman[41581]: 2024-03-26 01:57:11.758233168 +0000 UTC m=+0.108174825 container create b7a640456d42468b40baf16dd7f8bfbda93725d767c65616dc70f97e2a0b9251 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-26T01:57:12.008 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:11 smithi072 podman[41581]: 2024-03-26 01:57:11.668134233 +0000 UTC m=+0.018075847 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T01:57:12.009 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:11 smithi072 podman[41581]: 2024-03-26 01:57:11.970961537 +0000 UTC m=+0.320903152 container init b7a640456d42468b40baf16dd7f8bfbda93725d767c65616dc70f97e2a0b9251 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2) 2024-03-26T01:57:12.009 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:11 smithi072 podman[41581]: 2024-03-26 01:57:11.976865035 +0000 UTC m=+0.326806647 container start b7a640456d42468b40baf16dd7f8bfbda93725d767c65616dc70f97e2a0b9251 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-c, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-26T01:57:12.009 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T01:57:12.009 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 7 2024-03-26T01:57:12.009 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: pidfile_write: ignore empty --pid-file 2024-03-26T01:57:12.009 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: load: jerasure load: lrc 2024-03-26T01:57:12.009 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: RocksDB version: 7.9.2 2024-03-26T01:57:12.009 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Git sha 0 2024-03-26T01:57:12.009 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T01:57:12.009 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: DB SUMMARY 2024-03-26T01:57:12.009 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: DB Session ID: 55NYSVZOAL1RRNAVLZVX 2024-03-26T01:57:12.010 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: CURRENT file: CURRENT 2024-03-26T01:57:12.010 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T01:57:12.010 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-26T01:57:12.010 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-26T01:57:12.010 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-03-26T01:57:12.010 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.error_if_exists: 0 2024-03-26T01:57:12.010 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.create_if_missing: 0 2024-03-26T01:57:12.010 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.paranoid_checks: 1 2024-03-26T01:57:12.010 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T01:57:12.010 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T01:57:12.010 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T01:57:12.010 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.env: 0x562448872ae0 2024-03-26T01:57:12.010 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T01:57:12.010 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.info_log: 0x56244a53d920 2024-03-26T01:57:12.011 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T01:57:12.011 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.statistics: (nil) 2024-03-26T01:57:12.011 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.use_fsync: 0 2024-03-26T01:57:12.011 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.max_log_file_size: 0 2024-03-26T01:57:12.011 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T01:57:12.011 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T01:57:12.011 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T01:57:12.011 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T01:57:12.011 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.allow_fallocate: 1 2024-03-26T01:57:12.011 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T01:57:12.011 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T01:57:12.011 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.use_direct_reads: 0 2024-03-26T01:57:12.011 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T01:57:12.011 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T01:57:12.012 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.db_log_dir: 2024-03-26T01:57:12.012 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.wal_dir: 2024-03-26T01:57:12.012 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T01:57:12.012 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T01:57:12.012 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T01:57:12.012 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T01:57:12.012 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T01:57:12.012 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T01:57:12.012 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T01:57:12.012 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T01:57:12.012 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.write_buffer_manager: 0x56244a5101e0 2024-03-26T01:57:12.012 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T01:57:12.012 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T01:57:12.012 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T01:57:12.012 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T01:57:12.014 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T01:57:12.014 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T01:57:12.014 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T01:57:12.014 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T01:57:12.014 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.unordered_write: 0 2024-03-26T01:57:12.014 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T01:57:12.014 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T01:57:12.014 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T01:57:12.014 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T01:57:12.015 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.row_cache: None 2024-03-26T01:57:12.015 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.wal_filter: None 2024-03-26T01:57:12.015 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T01:57:12.015 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T01:57:12.015 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.two_write_queues: 0 2024-03-26T01:57:12.015 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T01:57:12.015 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.wal_compression: 0 2024-03-26T01:57:12.015 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.atomic_flush: 0 2024-03-26T01:57:12.015 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T01:57:12.015 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T01:57:12.015 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T01:57:12.015 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.log_readahead_size: 0 2024-03-26T01:57:12.015 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T01:57:12.015 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T01:57:12.015 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T01:57:12.015 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T01:57:12.016 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T01:57:12.016 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T01:57:12.016 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T01:57:12.016 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.max_background_jobs: 2 2024-03-26T01:57:12.016 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.max_background_compactions: -1 2024-03-26T01:57:12.016 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.max_subcompactions: 1 2024-03-26T01:57:12.016 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T01:57:12.016 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T01:57:12.016 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T01:57:12.016 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T01:57:12.016 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T01:57:12.016 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T01:57:12.016 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T01:57:12.016 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T01:57:12.016 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.max_open_files: -1 2024-03-26T01:57:12.016 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T01:57:12.017 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T01:57:12.017 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T01:57:12.017 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T01:57:12.017 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.max_background_flushes: -1 2024-03-26T01:57:12.017 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Compression algorithms supported: 2024-03-26T01:57:12.017 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T01:57:12.017 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: kZSTD supported: 0 2024-03-26T01:57:12.017 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: kXpressCompression supported: 0 2024-03-26T01:57:12.017 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T01:57:12.017 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: kZlibCompression supported: 1 2024-03-26T01:57:12.017 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: kSnappyCompression supported: 1 2024-03-26T01:57:12.017 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: kLZ4Compression supported: 1 2024-03-26T01:57:12.017 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: kBZip2Compression supported: 0 2024-03-26T01:57:12.017 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T01:57:12.017 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T01:57:12.017 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-26T01:57:12.018 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T01:57:12.018 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T01:57:12.018 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.merge_operator: 2024-03-26T01:57:12.018 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.compaction_filter: None 2024-03-26T01:57:12.018 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T01:57:12.018 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T01:57:12.018 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T01:57:12.018 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T01:57:12.018 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56244a53da20) 2024-03-26T01:57:12.018 INFO:journalctl@ceph.mon.c.smithi072.stdout: cache_index_and_filter_blocks: 1 2024-03-26T01:57:12.018 INFO:journalctl@ceph.mon.c.smithi072.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T01:57:12.018 INFO:journalctl@ceph.mon.c.smithi072.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T01:57:12.019 INFO:journalctl@ceph.mon.c.smithi072.stdout: pin_top_level_index_and_filter: 1 2024-03-26T01:57:12.019 INFO:journalctl@ceph.mon.c.smithi072.stdout: index_type: 0 2024-03-26T01:57:12.019 INFO:journalctl@ceph.mon.c.smithi072.stdout: data_block_index_type: 0 2024-03-26T01:57:12.019 INFO:journalctl@ceph.mon.c.smithi072.stdout: index_shortening: 1 2024-03-26T01:57:12.019 INFO:journalctl@ceph.mon.c.smithi072.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T01:57:12.019 INFO:journalctl@ceph.mon.c.smithi072.stdout: checksum: 4 2024-03-26T01:57:12.019 INFO:journalctl@ceph.mon.c.smithi072.stdout: no_block_cache: 0 2024-03-26T01:57:12.019 INFO:journalctl@ceph.mon.c.smithi072.stdout: block_cache: 0x56244a541090 2024-03-26T01:57:12.019 INFO:journalctl@ceph.mon.c.smithi072.stdout: block_cache_name: BinnedLRUCache 2024-03-26T01:57:12.020 INFO:journalctl@ceph.mon.c.smithi072.stdout: block_cache_options: 2024-03-26T01:57:12.020 INFO:journalctl@ceph.mon.c.smithi072.stdout: capacity : 536870912 2024-03-26T01:57:12.020 INFO:journalctl@ceph.mon.c.smithi072.stdout: num_shard_bits : 4 2024-03-26T01:57:12.020 INFO:journalctl@ceph.mon.c.smithi072.stdout: strict_capacity_limit : 0 2024-03-26T01:57:12.020 INFO:journalctl@ceph.mon.c.smithi072.stdout: high_pri_pool_ratio: 0.000 2024-03-26T01:57:12.020 INFO:journalctl@ceph.mon.c.smithi072.stdout: block_cache_compressed: (nil) 2024-03-26T01:57:12.020 INFO:journalctl@ceph.mon.c.smithi072.stdout: persistent_cache: (nil) 2024-03-26T01:57:12.020 INFO:journalctl@ceph.mon.c.smithi072.stdout: block_size: 4096 2024-03-26T01:57:12.020 INFO:journalctl@ceph.mon.c.smithi072.stdout: block_size_deviation: 10 2024-03-26T01:57:12.020 INFO:journalctl@ceph.mon.c.smithi072.stdout: block_restart_interval: 16 2024-03-26T01:57:12.020 INFO:journalctl@ceph.mon.c.smithi072.stdout: index_block_restart_interval: 1 2024-03-26T01:57:12.020 INFO:journalctl@ceph.mon.c.smithi072.stdout: metadata_block_size: 4096 2024-03-26T01:57:12.020 INFO:journalctl@ceph.mon.c.smithi072.stdout: partition_filters: 0 2024-03-26T01:57:12.020 INFO:journalctl@ceph.mon.c.smithi072.stdout: use_delta_encoding: 1 2024-03-26T01:57:12.020 INFO:journalctl@ceph.mon.c.smithi072.stdout: filter_policy: bloomfilter 2024-03-26T01:57:12.021 INFO:journalctl@ceph.mon.c.smithi072.stdout: whole_key_filtering: 1 2024-03-26T01:57:12.021 INFO:journalctl@ceph.mon.c.smithi072.stdout: verify_compression: 0 2024-03-26T01:57:12.021 INFO:journalctl@ceph.mon.c.smithi072.stdout: read_amp_bytes_per_bit: 0 2024-03-26T01:57:12.021 INFO:journalctl@ceph.mon.c.smithi072.stdout: format_version: 5 2024-03-26T01:57:12.021 INFO:journalctl@ceph.mon.c.smithi072.stdout: enable_index_compression: 1 2024-03-26T01:57:12.021 INFO:journalctl@ceph.mon.c.smithi072.stdout: block_align: 0 2024-03-26T01:57:12.021 INFO:journalctl@ceph.mon.c.smithi072.stdout: max_auto_readahead_size: 262144 2024-03-26T01:57:12.021 INFO:journalctl@ceph.mon.c.smithi072.stdout: prepopulate_block_cache: 0 2024-03-26T01:57:12.021 INFO:journalctl@ceph.mon.c.smithi072.stdout: initial_auto_readahead_size: 8192 2024-03-26T01:57:12.021 INFO:journalctl@ceph.mon.c.smithi072.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T01:57:12.021 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T01:57:12.021 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T01:57:12.021 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.compression: NoCompression 2024-03-26T01:57:12.021 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T01:57:12.022 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T01:57:12.022 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T01:57:12.022 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.num_levels: 7 2024-03-26T01:57:12.022 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T01:57:12.022 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T01:57:12.022 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T01:57:12.022 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T01:57:12.022 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T01:57:12.022 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T01:57:12.022 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T01:57:12.022 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T01:57:12.022 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T01:57:12.022 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T01:57:12.022 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T01:57:12.023 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T01:57:12.023 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T01:57:12.023 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T01:57:12.023 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T01:57:12.023 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T01:57:12.023 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T01:57:12.023 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T01:57:12.023 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T01:57:12.023 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T01:57:12.023 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T01:57:12.023 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T01:57:12.023 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T01:57:12.023 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T01:57:12.023 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T01:57:12.023 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T01:57:12.024 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T01:57:12.024 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T01:57:12.024 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T01:57:12.024 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T01:57:12.024 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T01:57:12.024 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T01:57:12.024 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T01:57:12.024 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T01:57:12.024 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T01:57:12.024 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T01:57:12.024 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T01:57:12.024 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T01:57:12.024 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T01:57:12.024 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T01:57:12.024 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T01:57:12.024 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T01:57:12.025 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T01:57:12.025 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T01:57:12.025 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T01:57:12.025 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T01:57:12.025 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T01:57:12.025 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T01:57:12.025 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T01:57:12.025 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T01:57:12.025 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T01:57:12.025 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T01:57:12.025 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T01:57:12.025 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.table_properties_collectors: 2024-03-26T01:57:12.025 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.inplace_update_support: 0 2024-03-26T01:57:12.026 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T01:57:12.026 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T01:57:12.026 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T01:57:12.026 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T01:57:12.026 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.bloom_locality: 0 2024-03-26T01:57:12.026 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.max_successive_merges: 0 2024-03-26T01:57:12.026 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T01:57:12.026 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T01:57:12.026 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T01:57:12.027 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T01:57:12.027 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.ttl: 2592000 2024-03-26T01:57:12.027 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T01:57:12.027 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T01:57:12.027 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T01:57:12.027 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.enable_blob_files: false 2024-03-26T01:57:12.027 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.min_blob_size: 0 2024-03-26T01:57:12.027 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T01:57:12.027 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T01:57:12.027 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T01:57:12.027 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T01:57:12.027 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T01:57:12.027 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T01:57:12.027 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T01:57:12.028 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T01:57:12.028 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T01:57:12.028 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-26T01:57:12.028 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 79436930-f8c4-4a98-b727-df84ea0343db 2024-03-26T01:57:12.028 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418232006019, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-26T01:57:12.028 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-26T01:57:12.028 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418232006407, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711418232, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "79436930-f8c4-4a98-b727-df84ea0343db", "db_session_id": "55NYSVZOAL1RRNAVLZVX", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-26T01:57:12.028 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711418232006486, "job": 1, "event": "recovery_finished"} 2024-03-26T01:57:12.028 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-26T01:57:12.028 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T01:57:12.028 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56244a632000 2024-03-26T01:57:12.029 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: DB pointer 0x56244a61c000 2024-03-26T01:57:12.029 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T01:57:12.029 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T01:57:12.029 INFO:journalctl@ceph.mon.c.smithi072.stdout: ** DB Stats ** 2024-03-26T01:57:12.029 INFO:journalctl@ceph.mon.c.smithi072.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T01:57:12.029 INFO:journalctl@ceph.mon.c.smithi072.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T01:57:12.029 INFO:journalctl@ceph.mon.c.smithi072.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T01:57:12.029 INFO:journalctl@ceph.mon.c.smithi072.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T01:57:12.029 INFO:journalctl@ceph.mon.c.smithi072.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T01:57:12.029 INFO:journalctl@ceph.mon.c.smithi072.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T01:57:12.029 INFO:journalctl@ceph.mon.c.smithi072.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T01:57:12.029 INFO:journalctl@ceph.mon.c.smithi072.stdout: 2024-03-26T01:57:12.029 INFO:journalctl@ceph.mon.c.smithi072.stdout: ** Compaction Stats [default] ** 2024-03-26T01:57:12.029 INFO:journalctl@ceph.mon.c.smithi072.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T01:57:12.029 INFO:journalctl@ceph.mon.c.smithi072.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T01:57:12.030 INFO:journalctl@ceph.mon.c.smithi072.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T01:57:12.030 INFO:journalctl@ceph.mon.c.smithi072.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T01:57:12.030 INFO:journalctl@ceph.mon.c.smithi072.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T01:57:12.030 INFO:journalctl@ceph.mon.c.smithi072.stdout: 2024-03-26T01:57:12.030 INFO:journalctl@ceph.mon.c.smithi072.stdout: ** Compaction Stats [default] ** 2024-03-26T01:57:12.030 INFO:journalctl@ceph.mon.c.smithi072.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T01:57:12.030 INFO:journalctl@ceph.mon.c.smithi072.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T01:57:12.030 INFO:journalctl@ceph.mon.c.smithi072.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.5 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T01:57:12.030 INFO:journalctl@ceph.mon.c.smithi072.stdout: 2024-03-26T01:57:12.030 INFO:journalctl@ceph.mon.c.smithi072.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T01:57:12.030 INFO:journalctl@ceph.mon.c.smithi072.stdout: 2024-03-26T01:57:12.030 INFO:journalctl@ceph.mon.c.smithi072.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T01:57:12.030 INFO:journalctl@ceph.mon.c.smithi072.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T01:57:12.030 INFO:journalctl@ceph.mon.c.smithi072.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T01:57:12.030 INFO:journalctl@ceph.mon.c.smithi072.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T01:57:12.031 INFO:journalctl@ceph.mon.c.smithi072.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T01:57:12.031 INFO:journalctl@ceph.mon.c.smithi072.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T01:57:12.031 INFO:journalctl@ceph.mon.c.smithi072.stdout: Cumulative compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T01:57:12.031 INFO:journalctl@ceph.mon.c.smithi072.stdout: Interval compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T01:57:12.031 INFO:journalctl@ceph.mon.c.smithi072.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T01:57:12.031 INFO:journalctl@ceph.mon.c.smithi072.stdout: Block cache BinnedLRUCache@0x56244a541090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-26T01:57:12.031 INFO:journalctl@ceph.mon.c.smithi072.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-26T01:57:12.031 INFO:journalctl@ceph.mon.c.smithi072.stdout: 2024-03-26T01:57:12.031 INFO:journalctl@ceph.mon.c.smithi072.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T01:57:12.031 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-26T01:57:12.031 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: using public_addrv [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] 2024-03-26T01:57:12.031 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: starting mon.c rank -1 at public addrs [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] at bind addrs [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T01:57:12.031 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: mon.c@-1(???) e0 preinit fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T01:57:12.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: mon.c@-1(synchronizing).mds e1 new map 2024-03-26T01:57:12.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-26T01:57:12.469 INFO:journalctl@ceph.mon.c.smithi072.stdout: e1 2024-03-26T01:57:12.469 INFO:journalctl@ceph.mon.c.smithi072.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T01:57:12.469 INFO:journalctl@ceph.mon.c.smithi072.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-26T01:57:12.469 INFO:journalctl@ceph.mon.c.smithi072.stdout: legacy client fscid: -1 2024-03-26T01:57:12.469 INFO:journalctl@ceph.mon.c.smithi072.stdout: 2024-03-26T01:57:12.469 INFO:journalctl@ceph.mon.c.smithi072.stdout: No filesystems configured 2024-03-26T01:57:12.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-26T01:57:12.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-26T01:57:12.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-26T01:57:12.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-26T01:57:12.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-26T01:57:12.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-26T01:57:12.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T01:57:12.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T01:57:12.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T01:57:12.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T01:57:12.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2963326343' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T01:57:12.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:12.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2963326343' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T01:57:12.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T01:57:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:57:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-26T01:57:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: Updating smithi173:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:57:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:57:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: Updating smithi173:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T01:57:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:12.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:12.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:12.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:12.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:12.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:12.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:12.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:57:12.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:12.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi072:172.21.15.72=a;smithi072:[v2:172.21.15.72:3301,v1:172.21.15.72:6790]=c;smithi173:172.21.15.173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:12.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: Saving service mon spec with placement smithi072:172.21.15.72=a;smithi072:[v2:172.21.15.72:3301,v1:172.21.15.72:6790]=c;smithi173:172.21.15.173=b;count:3 2024-03-26T01:57:12.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: Deploying daemon mon.b on smithi173 2024-03-26T01:57:12.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:12.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:12.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.173:0/175388125' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T01:57:12.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:57:12.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: mon.a calling monitor election 2024-03-26T01:57:12.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:57:12.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:57:12.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:12.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:57:12.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: mon.b calling monitor election 2024-03-26T01:57:12.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:57:12.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:12.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:57:12.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:57:12.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-26T01:57:12.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: monmap e2: 2 mons at {a=[v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T01:57:12.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: fsmap 2024-03-26T01:57:12.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T01:57:12.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: mgrmap e13: y(active, since 41s) 2024-03-26T01:57:12.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: overall HEALTH_OK 2024-03-26T01:57:12.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:12.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:12.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:12.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:57:12.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:12.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: Deploying daemon mon.c on smithi072 2024-03-26T01:57:12.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:12.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.173:0/265850551' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T01:57:12.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:57:12.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 ceph-mon[41619]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-26T01:57:12.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 bash[41581]: b7a640456d42468b40baf16dd7f8bfbda93725d767c65616dc70f97e2a0b9251 2024-03-26T01:57:12.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:12 smithi072 systemd[1]: Started Ceph mon.c for c86fbdd4-eb13-11ee-95d0-87774f69a715. 2024-03-26T01:57:17.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:57:17.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[30177]: mon.a calling monitor election 2024-03-26T01:57:17.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:57:17.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:57:17.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[30177]: mon.b calling monitor election 2024-03-26T01:57:17.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:57:17.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[30177]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:17.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:57:17.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[30177]: mon.c calling monitor election 2024-03-26T01:57:17.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:57:17.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[30177]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:17.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:57:17.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:57:17.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[30177]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T01:57:17.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[30177]: monmap e3: 3 mons at {a=[v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0],c=[v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T01:57:17.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[30177]: fsmap 2024-03-26T01:57:17.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[30177]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T01:57:17.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[30177]: mgrmap e13: y(active, since 49s) 2024-03-26T01:57:17.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[30177]: overall HEALTH_OK 2024-03-26T01:57:17.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:17.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:17.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:17.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:17.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:57:17.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[41619]: mon.a calling monitor election 2024-03-26T01:57:17.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:57:17.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:57:17.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[41619]: mon.b calling monitor election 2024-03-26T01:57:17.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:57:17.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[41619]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:17.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:57:17.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[41619]: mon.c calling monitor election 2024-03-26T01:57:17.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:57:17.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[41619]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:17.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:57:17.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:57:17.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[41619]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T01:57:17.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[41619]: monmap e3: 3 mons at {a=[v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0],c=[v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T01:57:17.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[41619]: fsmap 2024-03-26T01:57:17.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[41619]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T01:57:17.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[41619]: mgrmap e13: y(active, since 49s) 2024-03-26T01:57:17.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[41619]: overall HEALTH_OK 2024-03-26T01:57:17.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:17.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:17.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:17.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:17 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:17.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:17 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:57:17.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:17 smithi173 ceph-mon[33839]: mon.a calling monitor election 2024-03-26T01:57:17.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:17 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:57:17.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:17 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:57:17.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:17 smithi173 ceph-mon[33839]: mon.b calling monitor election 2024-03-26T01:57:17.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:17 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:57:17.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:17 smithi173 ceph-mon[33839]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:17.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:17 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:57:17.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:17 smithi173 ceph-mon[33839]: mon.c calling monitor election 2024-03-26T01:57:17.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:17 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:57:17.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:17 smithi173 ceph-mon[33839]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:17.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:17 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:57:17.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:17 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:57:17.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:17 smithi173 ceph-mon[33839]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T01:57:17.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:17 smithi173 ceph-mon[33839]: monmap e3: 3 mons at {a=[v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0],b=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0],c=[v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-26T01:57:17.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:17 smithi173 ceph-mon[33839]: fsmap 2024-03-26T01:57:17.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:17 smithi173 ceph-mon[33839]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T01:57:17.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:17 smithi173 ceph-mon[33839]: mgrmap e13: y(active, since 49s) 2024-03-26T01:57:17.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:17 smithi173 ceph-mon[33839]: overall HEALTH_OK 2024-03-26T01:57:17.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:17 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:17.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:17 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:17.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:17 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:17.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:17 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:18.312 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:18 smithi173 ceph-mon[33839]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:18.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:18 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:18.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:18 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:18.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:18 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:57:18.407 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.407+0000 7f76be496700 1 -- 172.21.15.173:0/3189175514 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76a0006240 msgr2=0x7f76a00086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:18.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.407+0000 7f76be496700 1 --2- 172.21.15.173:0/3189175514 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76a0006240 0x7f76a00086b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f76b8069a80 tx=0x7f76a800fe90 comp rx=0 tx=0).stop 2024-03-26T01:57:18.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.407+0000 7f76be496700 1 -- 172.21.15.173:0/3189175514 shutdown_connections 2024-03-26T01:57:18.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.407+0000 7f76be496700 1 --2- 172.21.15.173:0/3189175514 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76a0006240 0x7f76a00086b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:18.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.407+0000 7f76be496700 1 -- 172.21.15.173:0/3189175514 >> 172.21.15.173:0/3189175514 conn(0x7f76b8077170 msgr2=0x7f76b8077580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:57:18.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.407+0000 7f76be496700 1 -- 172.21.15.173:0/3189175514 shutdown_connections 2024-03-26T01:57:18.408 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.407+0000 7f76be496700 1 -- 172.21.15.173:0/3189175514 wait complete. 2024-03-26T01:57:18.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.408+0000 7f76be496700 1 Processor -- start 2024-03-26T01:57:18.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.408+0000 7f76be496700 1 -- start start 2024-03-26T01:57:18.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.408+0000 7f76be496700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76a0006240 0x7f76b8196de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:57:18.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.408+0000 7f76be496700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f76b8197320 0x7f76b819b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:57:18.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.408+0000 7f76be496700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76b819bbf0 0x7f76b819c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:57:18.411 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.409+0000 7f76be496700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b8197a10 con 0x7f76b819bbf0 2024-03-26T01:57:18.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.409+0000 7f76be496700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76b81a3960 con 0x7f76a0006240 2024-03-26T01:57:18.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.409+0000 7f76be496700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76b81a3aa0 con 0x7f76b8197320 2024-03-26T01:57:18.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.409+0000 7f76b7fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76a0006240 0x7f76b8196de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:57:18.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.409+0000 7f76b7fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76a0006240 0x7f76b8196de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:46448/0 (socket says 172.21.15.173:46448) 2024-03-26T01:57:18.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.409+0000 7f76bca33700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76b819bbf0 0x7f76b819c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:57:18.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.409+0000 7f76b7fff700 1 -- 172.21.15.173:0/1653690009 learned_addr learned my addr 172.21.15.173:0/1653690009 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-26T01:57:18.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.409+0000 7f76b77fe700 1 --2- 172.21.15.173:0/1653690009 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f76b8197320 0x7f76b819b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:57:18.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.409+0000 7f76b7fff700 1 -- 172.21.15.173:0/1653690009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76a0006240 msgr2=0x7f76b8196de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-26T01:57:18.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.409+0000 7f76b7fff700 1 -- 172.21.15.173:0/1653690009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76a0006240 msgr2=0x7f76b8196de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-26T01:57:18.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.409+0000 7f76b7fff700 1 --2- 172.21.15.173:0/1653690009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76a0006240 0x7f76b8196de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-26T01:57:18.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.409+0000 7f76b7fff700 1 --2- 172.21.15.173:0/1653690009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76a0006240 0x7f76b8196de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T01:57:18.412 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.409+0000 7f76b77fe700 1 -- 172.21.15.173:0/1653690009 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f76b8197320 msgr2=0x7f76b819b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-26T01:57:18.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.409+0000 7f76b77fe700 1 -- 172.21.15.173:0/1653690009 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f76b8197320 msgr2=0x7f76b819b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-26T01:57:18.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.409+0000 7f76b77fe700 1 --2- 172.21.15.173:0/1653690009 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f76b8197320 0x7f76b819b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-26T01:57:18.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.409+0000 7f76b77fe700 1 --2- 172.21.15.173:0/1653690009 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f76b8197320 0x7f76b819b6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T01:57:18.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.409+0000 7f76bca33700 1 -- 172.21.15.173:0/1653690009 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f76b8197320 msgr2=0x7f76b819b6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T01:57:18.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.409+0000 7f76bca33700 1 --2- 172.21.15.173:0/1653690009 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f76b8197320 0x7f76b819b6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:18.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.409+0000 7f76bca33700 1 -- 172.21.15.173:0/1653690009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76a0006240 msgr2=0x7f76b8196de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T01:57:18.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.409+0000 7f76bca33700 1 --2- 172.21.15.173:0/1653690009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76a0006240 0x7f76b8196de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:18.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.409+0000 7f76bca33700 1 -- 172.21.15.173:0/1653690009 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a800f8e0 con 0x7f76b819bbf0 2024-03-26T01:57:18.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.409+0000 7f76bca33700 1 --2- 172.21.15.173:0/1653690009 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76b819bbf0 0x7f76b819c0a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f76b000bdf0 tx=0x7f76b0009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:57:18.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.409+0000 7f76b57fa700 1 -- 172.21.15.173:0/1653690009 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b000cd50 con 0x7f76b819bbf0 2024-03-26T01:57:18.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.409+0000 7f76b57fa700 1 -- 172.21.15.173:0/1653690009 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f76b00148f0 con 0x7f76b819bbf0 2024-03-26T01:57:18.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.409+0000 7f76be496700 1 -- 172.21.15.173:0/1653690009 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76b81a3d80 con 0x7f76b819bbf0 2024-03-26T01:57:18.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.409+0000 7f76b57fa700 1 -- 172.21.15.173:0/1653690009 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b00127d0 con 0x7f76b819bbf0 2024-03-26T01:57:18.413 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.410+0000 7f76be496700 1 -- 172.21.15.173:0/1653690009 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76b81a4220 con 0x7f76b819bbf0 2024-03-26T01:57:18.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.410+0000 7f76b57fa700 1 -- 172.21.15.173:0/1653690009 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) v1 ==== 45949+0+0 (secure 0 0 0) 0x7f76b0012970 con 0x7f76b819bbf0 2024-03-26T01:57:18.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.410+0000 7f76be496700 1 -- 172.21.15.173:0/1653690009 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76b8190ff0 con 0x7f76b819bbf0 2024-03-26T01:57:18.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.410+0000 7f76b57fa700 1 --2- 172.21.15.173:0/1653690009 >> v2:172.21.15.72:6800/2923394687 conn(0x7f76a40393d0 0x7f76a403b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:57:18.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.411+0000 7f76b57fa700 1 -- 172.21.15.173:0/1653690009 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (secure 0 0 0) 0x7f76b004b7e0 con 0x7f76b819bbf0 2024-03-26T01:57:18.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.411+0000 7f76b7fff700 1 --2- 172.21.15.173:0/1653690009 >> v2:172.21.15.72:6800/2923394687 conn(0x7f76a40393d0 0x7f76a403b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:57:18.414 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.411+0000 7f76b7fff700 1 --2- 172.21.15.173:0/1653690009 >> v2:172.21.15.72:6800/2923394687 conn(0x7f76a40393d0 0x7f76a403b890 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f76a8005e00 tx=0x7f76a800f4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:57:18.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.416+0000 7f76b57fa700 1 -- 172.21.15.173:0/1653690009 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f76b001d3b0 con 0x7f76b819bbf0 2024-03-26T01:57:18.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:18 smithi072 ceph-mon[30177]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:18.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:18 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:18.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:18 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:18.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:18 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:57:18.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:18 smithi072 ceph-mon[41619]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:18.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:18 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:18.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:18 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:18.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:18 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:57:18.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.702+0000 7f76be496700 1 -- 172.21.15.173:0/1653690009 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f76b8198140 con 0x7f76b819bbf0 2024-03-26T01:57:18.703 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.703+0000 7f76b57fa700 1 -- 172.21.15.173:0/1653690009 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1281 (secure 0 0 0) 0x7f76b0019030 con 0x7f76b819bbf0 2024-03-26T01:57:18.704 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-03-26T01:57:18.704 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":3,"fsid":"c86fbdd4-eb13-11ee-95d0-87774f69a715","modified":"2024-03-26T01:57:12.034021Z","created":"2024-03-26T01:55:12.142924Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:3300","nonce":0},{"type":"v1","addr":"172.21.15.72:6789","nonce":0}]},"addr":"172.21.15.72:6789/0","public_addr":"172.21.15.72:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:3301","nonce":0},{"type":"v1","addr":"172.21.15.72:6790","nonce":0}]},"addr":"172.21.15.72:6790/0","public_addr":"172.21.15.72:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-26T01:57:18.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.705+0000 7f76be496700 1 -- 172.21.15.173:0/1653690009 >> v2:172.21.15.72:6800/2923394687 conn(0x7f76a40393d0 msgr2=0x7f76a403b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:18.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.705+0000 7f76be496700 1 --2- 172.21.15.173:0/1653690009 >> v2:172.21.15.72:6800/2923394687 conn(0x7f76a40393d0 0x7f76a403b890 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f76a8005e00 tx=0x7f76a800f4a0 comp rx=0 tx=0).stop 2024-03-26T01:57:18.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.705+0000 7f76be496700 1 -- 172.21.15.173:0/1653690009 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76b819bbf0 msgr2=0x7f76b819c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:18.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.705+0000 7f76be496700 1 --2- 172.21.15.173:0/1653690009 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76b819bbf0 0x7f76b819c0a0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f76b000bdf0 tx=0x7f76b0009f70 comp rx=0 tx=0).stop 2024-03-26T01:57:18.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.705+0000 7f76be496700 1 -- 172.21.15.173:0/1653690009 shutdown_connections 2024-03-26T01:57:18.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.705+0000 7f76be496700 1 --2- 172.21.15.173:0/1653690009 >> v2:172.21.15.72:6800/2923394687 conn(0x7f76a40393d0 0x7f76a403b890 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:18.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.705+0000 7f76be496700 1 --2- 172.21.15.173:0/1653690009 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f76b819bbf0 0x7f76b819c0a0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:18.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.705+0000 7f76be496700 1 --2- 172.21.15.173:0/1653690009 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f76a0006240 0x7f76b8196de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:18.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.705+0000 7f76be496700 1 --2- 172.21.15.173:0/1653690009 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f76b8197320 0x7f76b819b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:18.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.705+0000 7f76be496700 1 -- 172.21.15.173:0/1653690009 >> 172.21.15.173:0/1653690009 conn(0x7f76b8077170 msgr2=0x7f76b818e130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:57:18.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.705+0000 7f76be496700 1 -- 172.21.15.173:0/1653690009 shutdown_connections 2024-03-26T01:57:18.706 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:18.706+0000 7f76be496700 1 -- 172.21.15.173:0/1653690009 wait complete. 2024-03-26T01:57:18.707 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 3 2024-03-26T01:57:19.397 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-26T01:57:19.398 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-26T01:57:19.718 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:19.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:19.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[41619]: Updating smithi072:/etc/ceph/ceph.conf 2024-03-26T01:57:19.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[41619]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-26T01:57:19.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.173:0/1653690009' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T01:57:19.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[41619]: Updating smithi072:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:57:19.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[41619]: Updating smithi173:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:57:19.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[41619]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:19.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[41619]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T01:57:19.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:57:19.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:57:19.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:19.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[41619]: Reconfiguring daemon mon.a on smithi072 2024-03-26T01:57:19.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:19.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:19.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[30177]: Updating smithi072:/etc/ceph/ceph.conf 2024-03-26T01:57:19.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[30177]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-26T01:57:19.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.173:0/1653690009' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T01:57:19.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[30177]: Updating smithi072:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:57:19.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[30177]: Updating smithi173:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:57:19.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[30177]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:19.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[30177]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T01:57:19.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:57:19.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:57:19.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:19.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:19 smithi072 ceph-mon[30177]: Reconfiguring daemon mon.a on smithi072 2024-03-26T01:57:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:19 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:19 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:19 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:19 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:19 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:19 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:19 smithi173 ceph-mon[33839]: Updating smithi072:/etc/ceph/ceph.conf 2024-03-26T01:57:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:19 smithi173 ceph-mon[33839]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-26T01:57:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.173:0/1653690009' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T01:57:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:19 smithi173 ceph-mon[33839]: Updating smithi072:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:57:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:19 smithi173 ceph-mon[33839]: Updating smithi173:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T01:57:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:19 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:19 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:19 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:19 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:19 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:19 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:19 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:19 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:19 smithi173 ceph-mon[33839]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:19.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:19 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:19.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:19 smithi173 ceph-mon[33839]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T01:57:19.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:19 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:57:19.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:19 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:57:19.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:19 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:19.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:19 smithi173 ceph-mon[33839]: Reconfiguring daemon mon.a on smithi072 2024-03-26T01:57:20.216 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:57:22.151 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:21 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:22.151 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:21 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:22.151 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:21 smithi072 ceph-mon[30177]: Reconfiguring mon.c (monmap changed)... 2024-03-26T01:57:22.151 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:21 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:57:22.151 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:21 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:57:22.151 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:21 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:22.152 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:21 smithi072 ceph-mon[30177]: Reconfiguring daemon mon.c on smithi072 2024-03-26T01:57:22.152 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:21 smithi072 ceph-mon[30177]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:22.152 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:21 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:22.152 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:21 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:22.152 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:21 smithi072 ceph-mon[41619]: Reconfiguring mon.c (monmap changed)... 2024-03-26T01:57:22.152 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:21 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:57:22.152 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:21 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:57:22.152 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:21 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:22.152 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:21 smithi072 ceph-mon[41619]: Reconfiguring daemon mon.c on smithi072 2024-03-26T01:57:22.152 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:21 smithi072 ceph-mon[41619]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:22.312 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:21 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:21 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:21 smithi173 ceph-mon[33839]: Reconfiguring mon.c (monmap changed)... 2024-03-26T01:57:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:21 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:57:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:21 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:57:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:21 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:21 smithi173 ceph-mon[33839]: Reconfiguring daemon mon.c on smithi072 2024-03-26T01:57:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:21 smithi173 ceph-mon[33839]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:23.180 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.180+0000 7fdd4161b700 1 -- 172.21.15.72:0/2586051214 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd3c1080d0 msgr2=0x7fdd3c1084b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:23.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.180+0000 7fdd4161b700 1 --2- 172.21.15.72:0/2586051214 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd3c1080d0 0x7fdd3c1084b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c00b0d0 tx=0x7fdd2c00b3e0 comp rx=0 tx=0).stop 2024-03-26T01:57:23.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.180+0000 7fdd4161b700 1 -- 172.21.15.72:0/2586051214 shutdown_connections 2024-03-26T01:57:23.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.180+0000 7fdd4161b700 1 --2- 172.21.15.72:0/2586051214 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd3c06a6d0 0x7fdd3c10ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:23.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.180+0000 7fdd4161b700 1 --2- 172.21.15.72:0/2586051214 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd3c1080d0 0x7fdd3c1084b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:23.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.180+0000 7fdd4161b700 1 --2- 172.21.15.72:0/2586051214 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdd3c069d30 0x7fdd3c06a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:23.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.180+0000 7fdd4161b700 1 -- 172.21.15.72:0/2586051214 >> 172.21.15.72:0/2586051214 conn(0x7fdd3c077eb0 msgr2=0x7fdd3c0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:57:23.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.180+0000 7fdd4161b700 1 -- 172.21.15.72:0/2586051214 shutdown_connections 2024-03-26T01:57:23.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.180+0000 7fdd4161b700 1 -- 172.21.15.72:0/2586051214 wait complete. 2024-03-26T01:57:23.181 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.181+0000 7fdd4161b700 1 Processor -- start 2024-03-26T01:57:23.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.181+0000 7fdd4161b700 1 -- start start 2024-03-26T01:57:23.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.181+0000 7fdd4161b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd3c069d30 0x7fdd3c197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:57:23.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.181+0000 7fdd4161b700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd3c06a6d0 0x7fdd3c198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:57:23.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.181+0000 7fdd4161b700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdd3c1080d0 0x7fdd3c19c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:57:23.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.181+0000 7fdd4161b700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c19cc90 con 0x7fdd3c06a6d0 2024-03-26T01:57:23.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.181+0000 7fdd4161b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c19ce00 con 0x7fdd3c069d30 2024-03-26T01:57:23.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.181+0000 7fdd4161b700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3c19cfc0 con 0x7fdd3c1080d0 2024-03-26T01:57:23.183 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.181+0000 7fdd3a7fc700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd3c06a6d0 0x7fdd3c198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:57:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.181+0000 7fdd3a7fc700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd3c06a6d0 0x7fdd3c198340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49256/0 (socket says 172.21.15.72:49256) 2024-03-26T01:57:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.181+0000 7fdd3b7fe700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdd3c1080d0 0x7fdd3c19c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:57:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.181+0000 7fdd3a7fc700 1 -- 172.21.15.72:0/930390837 learned_addr learned my addr 172.21.15.72:0/930390837 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:57:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.181+0000 7fdd3affd700 1 --2- 172.21.15.72:0/930390837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd3c069d30 0x7fdd3c197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:57:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.181+0000 7fdd3a7fc700 1 -- 172.21.15.72:0/930390837 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdd3c1080d0 msgr2=0x7fdd3c19c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.181+0000 7fdd3a7fc700 1 --2- 172.21.15.72:0/930390837 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdd3c1080d0 0x7fdd3c19c680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.181+0000 7fdd3a7fc700 1 -- 172.21.15.72:0/930390837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd3c069d30 msgr2=0x7fdd3c197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.181+0000 7fdd3a7fc700 1 --2- 172.21.15.72:0/930390837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd3c069d30 0x7fdd3c197e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.181+0000 7fdd3a7fc700 1 -- 172.21.15.72:0/930390837 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd2c009d00 con 0x7fdd3c06a6d0 2024-03-26T01:57:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.182+0000 7fdd3a7fc700 1 --2- 172.21.15.72:0/930390837 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd3c06a6d0 0x7fdd3c198340 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdd2400e9a0 tx=0x7fdd2400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:57:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.182+0000 7fdd33fff700 1 -- 172.21.15.72:0/930390837 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd24009790 con 0x7fdd3c06a6d0 2024-03-26T01:57:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.182+0000 7fdd33fff700 1 -- 172.21.15.72:0/930390837 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fdd24010430 con 0x7fdd3c06a6d0 2024-03-26T01:57:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.182+0000 7fdd4161b700 1 -- 172.21.15.72:0/930390837 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd3c19d2a0 con 0x7fdd3c06a6d0 2024-03-26T01:57:23.184 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.182+0000 7fdd33fff700 1 -- 172.21.15.72:0/930390837 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd24018680 con 0x7fdd3c06a6d0 2024-03-26T01:57:23.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.182+0000 7fdd4161b700 1 -- 172.21.15.72:0/930390837 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd3c1a4b00 con 0x7fdd3c06a6d0 2024-03-26T01:57:23.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.183+0000 7fdd33fff700 1 -- 172.21.15.72:0/930390837 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) v1 ==== 45949+0+0 (secure 0 0 0) 0x7fdd2400ca10 con 0x7fdd3c06a6d0 2024-03-26T01:57:23.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.183+0000 7fdd4161b700 1 -- 172.21.15.72:0/930390837 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd3c068d40 con 0x7fdd3c06a6d0 2024-03-26T01:57:23.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.183+0000 7fdd33fff700 1 --2- 172.21.15.72:0/930390837 >> v2:172.21.15.72:6800/2923394687 conn(0x7fdd28039180 0x7fdd2803b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:57:23.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.183+0000 7fdd33fff700 1 -- 172.21.15.72:0/930390837 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (secure 0 0 0) 0x7fdd24014070 con 0x7fdd3c06a6d0 2024-03-26T01:57:23.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.183+0000 7fdd3affd700 1 --2- 172.21.15.72:0/930390837 >> v2:172.21.15.72:6800/2923394687 conn(0x7fdd28039180 0x7fdd2803b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:57:23.185 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.183+0000 7fdd3affd700 1 --2- 172.21.15.72:0/930390837 >> v2:172.21.15.72:6800/2923394687 conn(0x7fdd28039180 0x7fdd2803b640 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c009fd0 tx=0x7fdd2c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:57:23.188 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.188+0000 7fdd33fff700 1 -- 172.21.15.72:0/930390837 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fdd24010880 con 0x7fdd3c06a6d0 2024-03-26T01:57:23.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.381+0000 7fdd4161b700 1 -- 172.21.15.72:0/930390837 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fdd3c199110 con 0x7fdd3c06a6d0 2024-03-26T01:57:23.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.382+0000 7fdd33fff700 1 -- 172.21.15.72:0/930390837 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+273 (secure 0 0 0) 0x7fdd24025170 con 0x7fdd3c06a6d0 2024-03-26T01:57:23.382 INFO:teuthology.orchestra.run.smithi072.stdout:# minimal ceph.conf for c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T01:57:23.382 INFO:teuthology.orchestra.run.smithi072.stdout:[global] 2024-03-26T01:57:23.382 INFO:teuthology.orchestra.run.smithi072.stdout: fsid = c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T01:57:23.382 INFO:teuthology.orchestra.run.smithi072.stdout: mon_host = [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] 2024-03-26T01:57:23.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.384+0000 7fdd4161b700 1 -- 172.21.15.72:0/930390837 >> v2:172.21.15.72:6800/2923394687 conn(0x7fdd28039180 msgr2=0x7fdd2803b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:23.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.384+0000 7fdd4161b700 1 --2- 172.21.15.72:0/930390837 >> v2:172.21.15.72:6800/2923394687 conn(0x7fdd28039180 0x7fdd2803b640 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c009fd0 tx=0x7fdd2c00b040 comp rx=0 tx=0).stop 2024-03-26T01:57:23.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.384+0000 7fdd4161b700 1 -- 172.21.15.72:0/930390837 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd3c06a6d0 msgr2=0x7fdd3c198340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:23.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.384+0000 7fdd4161b700 1 --2- 172.21.15.72:0/930390837 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd3c06a6d0 0x7fdd3c198340 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdd2400e9a0 tx=0x7fdd2400ecb0 comp rx=0 tx=0).stop 2024-03-26T01:57:23.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.384+0000 7fdd4161b700 1 -- 172.21.15.72:0/930390837 shutdown_connections 2024-03-26T01:57:23.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.384+0000 7fdd4161b700 1 --2- 172.21.15.72:0/930390837 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdd3c1080d0 0x7fdd3c19c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:23.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.384+0000 7fdd4161b700 1 --2- 172.21.15.72:0/930390837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdd3c069d30 0x7fdd3c197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:23.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.384+0000 7fdd4161b700 1 --2- 172.21.15.72:0/930390837 >> v2:172.21.15.72:6800/2923394687 conn(0x7fdd28039180 0x7fdd2803b640 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:23.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.384+0000 7fdd4161b700 1 --2- 172.21.15.72:0/930390837 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdd3c06a6d0 0x7fdd3c198340 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:23.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.384+0000 7fdd4161b700 1 -- 172.21.15.72:0/930390837 >> 172.21.15.72:0/930390837 conn(0x7fdd3c077eb0 msgr2=0x7fdd3c105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:57:23.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.384+0000 7fdd4161b700 1 -- 172.21.15.72:0/930390837 shutdown_connections 2024-03-26T01:57:23.385 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:23.384+0000 7fdd4161b700 1 -- 172.21.15.72:0/930390837 wait complete. 2024-03-26T01:57:23.916 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-26T01:57:23.916 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-26T01:57:23.917 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T01:57:23.946 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-26T01:57:23.946 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:57:24.013 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-26T01:57:24.013 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T01:57:24.042 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-26T01:57:24.042 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T01:57:24.109 INFO:tasks.cephadm:Adding mgr.y on smithi072 2024-03-26T01:57:24.109 INFO:tasks.cephadm:Adding mgr.x on smithi173 2024-03-26T01:57:24.109 DEBUG:teuthology.orchestra.run.smithi173:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph orch apply mgr '2;smithi072=y;smithi173=x' 2024-03-26T01:57:24.173 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:23 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:24.173 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:23 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:24.173 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:23 smithi173 ceph-mon[33839]: Reconfiguring mon.b (monmap changed)... 2024-03-26T01:57:24.173 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:23 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:57:24.173 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:23 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:57:24.173 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:23 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:24.173 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:23 smithi173 ceph-mon[33839]: Reconfiguring daemon mon.b on smithi173 2024-03-26T01:57:24.173 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:23 smithi173 ceph-mon[33839]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:24.173 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:23 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/930390837' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:24.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:23 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:24.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:23 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:24.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:23 smithi072 ceph-mon[30177]: Reconfiguring mon.b (monmap changed)... 2024-03-26T01:57:24.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:23 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:57:24.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:23 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:57:24.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:23 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:24.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:23 smithi072 ceph-mon[30177]: Reconfiguring daemon mon.b on smithi173 2024-03-26T01:57:24.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:23 smithi072 ceph-mon[30177]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:24.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:23 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/930390837' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:24.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:23 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:24.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:23 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:24.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:23 smithi072 ceph-mon[41619]: Reconfiguring mon.b (monmap changed)... 2024-03-26T01:57:24.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:23 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T01:57:24.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:23 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T01:57:24.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:23 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:24.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:23 smithi072 ceph-mon[41619]: Reconfiguring daemon mon.b on smithi173 2024-03-26T01:57:24.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:23 smithi072 ceph-mon[41619]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:24.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:23 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/930390837' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:24.573 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T01:57:25.770 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:25 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:25.770 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:25 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:25.770 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:25 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:25.770 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:25 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:25.770 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:25 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:25.770 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:25 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:25.770 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:25 smithi173 ceph-mon[33839]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:25.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:25 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:25.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:25 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:25.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:25 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:25.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:25 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:25.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:25 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:25.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:25 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:25.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:25 smithi072 ceph-mon[30177]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:25.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:25 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:25.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:25 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:25.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:25 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:25.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:25 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:25.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:25 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:25.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:25 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:25.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:25 smithi072 ceph-mon[41619]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:26.122 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.109+0000 7f18cb0e8700 1 -- 172.21.15.173:0/3475147901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18c4102b50 msgr2=0x7f18c4109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:26.122 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.109+0000 7f18cb0e8700 1 --2- 172.21.15.173:0/3475147901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18c4102b50 0x7f18c4109400 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f18bc00b0d0 tx=0x7f18bc00b3e0 comp rx=0 tx=0).stop 2024-03-26T01:57:26.122 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.109+0000 7f18cb0e8700 1 -- 172.21.15.173:0/3475147901 shutdown_connections 2024-03-26T01:57:26.122 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.109+0000 7f18cb0e8700 1 --2- 172.21.15.173:0/3475147901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18c4102b50 0x7f18c4109400 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:26.122 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.109+0000 7f18cb0e8700 1 --2- 172.21.15.173:0/3475147901 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f18c4100fb0 0x7f18c41013d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:26.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.109+0000 7f18cb0e8700 1 --2- 172.21.15.173:0/3475147901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18c41021b0 0x7f18c4102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:26.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.109+0000 7f18cb0e8700 1 -- 172.21.15.173:0/3475147901 >> 172.21.15.173:0/3475147901 conn(0x7f18c40fc550 msgr2=0x7f18c40fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:57:26.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.109+0000 7f18cb0e8700 1 -- 172.21.15.173:0/3475147901 shutdown_connections 2024-03-26T01:57:26.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.109+0000 7f18cb0e8700 1 -- 172.21.15.173:0/3475147901 wait complete. 2024-03-26T01:57:26.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.110+0000 7f18cb0e8700 1 Processor -- start 2024-03-26T01:57:26.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.110+0000 7f18cb0e8700 1 -- start start 2024-03-26T01:57:26.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.110+0000 7f18cb0e8700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f18c4100fb0 0x7f18c41978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:57:26.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.110+0000 7f18cb0e8700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18c41021b0 0x7f18c4197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:57:26.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.110+0000 7f18cb0e8700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18c4102b50 0x7f18c419ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:57:26.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.110+0000 7f18cb0e8700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c419f460 con 0x7f18c41021b0 2024-03-26T01:57:26.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.110+0000 7f18cb0e8700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18c419f5a0 con 0x7f18c4102b50 2024-03-26T01:57:26.123 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.110+0000 7f18cb0e8700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18c419f730 con 0x7f18c4100fb0 2024-03-26T01:57:26.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.110+0000 7f18c9685700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18c4102b50 0x7f18c419ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:57:26.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.110+0000 7f18c9685700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18c4102b50 0x7f18c419ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:46474/0 (socket says 172.21.15.173:46474) 2024-03-26T01:57:26.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.110+0000 7f18c9685700 1 -- 172.21.15.173:0/308555751 learned_addr learned my addr 172.21.15.173:0/308555751 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-26T01:57:26.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.110+0000 7f18c8e84700 1 --2- 172.21.15.173:0/308555751 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f18c4100fb0 0x7f18c41978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:57:26.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.110+0000 7f18c3fff700 1 --2- 172.21.15.173:0/308555751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18c41021b0 0x7f18c4197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:57:26.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.110+0000 7f18c9685700 1 -- 172.21.15.173:0/308555751 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f18c4100fb0 msgr2=0x7f18c41978d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:26.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.110+0000 7f18c9685700 1 --2- 172.21.15.173:0/308555751 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f18c4100fb0 0x7f18c41978d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:26.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.110+0000 7f18c9685700 1 -- 172.21.15.173:0/308555751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18c41021b0 msgr2=0x7f18c4197e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:26.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.110+0000 7f18c9685700 1 --2- 172.21.15.173:0/308555751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18c41021b0 0x7f18c4197e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:26.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.110+0000 7f18c9685700 1 -- 172.21.15.173:0/308555751 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18bc009d00 con 0x7f18c4102b50 2024-03-26T01:57:26.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.110+0000 7f18c8e84700 1 --2- 172.21.15.173:0/308555751 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f18c4100fb0 0x7f18c41978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T01:57:26.124 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.110+0000 7f18c9685700 1 --2- 172.21.15.173:0/308555751 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18c4102b50 0x7f18c419ee90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f18bc0077f0 tx=0x7f18bc0078d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:57:26.125 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.110+0000 7f18c1ffb700 1 -- 172.21.15.173:0/308555751 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18bc00e040 con 0x7f18c4102b50 2024-03-26T01:57:26.125 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.110+0000 7f18c1ffb700 1 -- 172.21.15.173:0/308555751 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f18bc005070 con 0x7f18c4102b50 2024-03-26T01:57:26.125 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.110+0000 7f18cb0e8700 1 -- 172.21.15.173:0/308555751 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c419f9b0 con 0x7f18c4102b50 2024-03-26T01:57:26.125 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.110+0000 7f18c1ffb700 1 -- 172.21.15.173:0/308555751 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18bc01b6c0 con 0x7f18c4102b50 2024-03-26T01:57:26.125 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.111+0000 7f18cb0e8700 1 -- 172.21.15.173:0/308555751 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c419fdc0 con 0x7f18c4102b50 2024-03-26T01:57:26.125 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.111+0000 7f18c1ffb700 1 -- 172.21.15.173:0/308555751 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 13) v1 ==== 45949+0+0 (secure 0 0 0) 0x7f18bc01b820 con 0x7f18c4102b50 2024-03-26T01:57:26.125 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.112+0000 7f18cb0e8700 1 -- 172.21.15.173:0/308555751 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c4108370 con 0x7f18c4102b50 2024-03-26T01:57:26.125 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.112+0000 7f18c1ffb700 1 --2- 172.21.15.173:0/308555751 >> v2:172.21.15.72:6800/2923394687 conn(0x7f18b4039180 0x7f18b403b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:57:26.125 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.112+0000 7f18c1ffb700 1 -- 172.21.15.173:0/308555751 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (secure 0 0 0) 0x7f18bc04cea0 con 0x7f18c4102b50 2024-03-26T01:57:26.125 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.112+0000 7f18c8e84700 1 --2- 172.21.15.173:0/308555751 >> v2:172.21.15.72:6800/2923394687 conn(0x7f18b4039180 0x7f18b403b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:57:26.125 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.112+0000 7f18c8e84700 1 --2- 172.21.15.173:0/308555751 >> v2:172.21.15.72:6800/2923394687 conn(0x7f18b4039180 0x7f18b403b640 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f18c4102010 tx=0x7f18b8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:57:26.125 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.117+0000 7f18c1ffb700 1 -- 172.21.15.173:0/308555751 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f18bc011920 con 0x7f18c4102b50 2024-03-26T01:57:26.319 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.319+0000 7f18cb0e8700 1 -- 172.21.15.173:0/308555751 --> v2:172.21.15.72:6800/2923394687 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi072=y;smithi173=x", "target": ["mon-mgr", ""]}) v1 -- 0x7f18c402d9e0 con 0x7f18b4039180 2024-03-26T01:57:26.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.326+0000 7f18c1ffb700 1 -- 172.21.15.173:0/308555751 <== mgr.14152 v2:172.21.15.72:6800/2923394687 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f18c402d9e0 con 0x7f18b4039180 2024-03-26T01:57:26.326 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mgr update... 2024-03-26T01:57:26.328 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.328+0000 7f18cb0e8700 1 -- 172.21.15.173:0/308555751 >> v2:172.21.15.72:6800/2923394687 conn(0x7f18b4039180 msgr2=0x7f18b403b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:26.328 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.328+0000 7f18cb0e8700 1 --2- 172.21.15.173:0/308555751 >> v2:172.21.15.72:6800/2923394687 conn(0x7f18b4039180 0x7f18b403b640 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f18c4102010 tx=0x7f18b8009260 comp rx=0 tx=0).stop 2024-03-26T01:57:26.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.328+0000 7f18cb0e8700 1 -- 172.21.15.173:0/308555751 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18c4102b50 msgr2=0x7f18c419ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:26.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.328+0000 7f18cb0e8700 1 --2- 172.21.15.173:0/308555751 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18c4102b50 0x7f18c419ee90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f18bc0077f0 tx=0x7f18bc0078d0 comp rx=0 tx=0).stop 2024-03-26T01:57:26.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.328+0000 7f18cb0e8700 1 -- 172.21.15.173:0/308555751 shutdown_connections 2024-03-26T01:57:26.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.328+0000 7f18cb0e8700 1 --2- 172.21.15.173:0/308555751 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18c4102b50 0x7f18c419ee90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:26.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.328+0000 7f18cb0e8700 1 --2- 172.21.15.173:0/308555751 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f18c4100fb0 0x7f18c41978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:26.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.328+0000 7f18cb0e8700 1 --2- 172.21.15.173:0/308555751 >> v2:172.21.15.72:6800/2923394687 conn(0x7f18b4039180 0x7f18b403b640 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:26.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.328+0000 7f18cb0e8700 1 --2- 172.21.15.173:0/308555751 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f18c41021b0 0x7f18c4197e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:26.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.328+0000 7f18cb0e8700 1 -- 172.21.15.173:0/308555751 >> 172.21.15.173:0/308555751 conn(0x7f18c40fc550 msgr2=0x7f18c41054b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:57:26.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.328+0000 7f18cb0e8700 1 -- 172.21.15.173:0/308555751 shutdown_connections 2024-03-26T01:57:26.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:57:26.328+0000 7f18cb0e8700 1 -- 172.21.15.173:0/308555751 wait complete. 2024-03-26T01:57:26.968 DEBUG:teuthology.orchestra.run.smithi173:mgr.x> sudo journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@mgr.x.service 2024-03-26T01:57:27.010 INFO:tasks.cephadm:Deploying OSDs... 2024-03-26T01:57:27.011 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-26T01:57:27.011 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T01:57:27.031 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T01:57:27.031 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_1 2024-03-26T01:57:27.088 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T01:57:27.089 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:57:27.089 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-03-26T01:57:27.089 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:57:27.089 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:57:27.089 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-03-26 01:57:11.490311009 +0000 2024-03-26T01:57:27.089 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-03-26 01:51:02.710512162 +0000 2024-03-26T01:57:27.089 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-03-26 01:51:02.710512162 +0000 2024-03-26T01:57:27.089 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-03-26 01:51:02.710512162 +0000 2024-03-26T01:57:27.089 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T01:57:27.158 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-03-26T01:57:27.158 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-03-26T01:57:27.158 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000275591 s, 1.9 MB/s 2024-03-26T01:57:27.159 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T01:57:27.216 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_2 2024-03-26T01:57:27.272 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T01:57:27.272 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:57:27.272 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-26T01:57:27.272 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:57:27.272 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:57:27.272 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-03-26 01:57:11.490311009 +0000 2024-03-26T01:57:27.273 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-03-26 01:51:03.046504895 +0000 2024-03-26T01:57:27.273 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-03-26 01:51:03.046504895 +0000 2024-03-26T01:57:27.273 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-03-26 01:51:03.046504895 +0000 2024-03-26T01:57:27.273 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T01:57:27.340 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-03-26T01:57:27.340 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-03-26T01:57:27.340 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000205179 s, 2.5 MB/s 2024-03-26T01:57:27.341 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T01:57:27.397 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_3 2024-03-26T01:57:27.453 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T01:57:27.454 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:57:27.454 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-26T01:57:27.454 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:57:27.454 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:57:27.454 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-03-26 01:57:11.490311009 +0000 2024-03-26T01:57:27.454 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-03-26 01:51:03.390497456 +0000 2024-03-26T01:57:27.455 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-03-26 01:51:03.390497456 +0000 2024-03-26T01:57:27.455 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-03-26 01:51:03.390497456 +0000 2024-03-26T01:57:27.455 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T01:57:27.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:27 smithi072 ceph-mon[30177]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi072=y;smithi173=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:27.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:27 smithi072 ceph-mon[30177]: Saving service mgr spec with placement smithi072=y;smithi173=x;count:2 2024-03-26T01:57:27.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:27 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:27.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:27 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:27.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:27 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:27 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:27 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:27 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:57:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:27 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T01:57:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:27 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:57:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:27 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:27 smithi072 ceph-mon[30177]: Deploying daemon mgr.x on smithi173 2024-03-26T01:57:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:27 smithi072 ceph-mon[30177]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:27.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:27 smithi072 ceph-mon[41619]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi072=y;smithi173=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:27.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:27 smithi072 ceph-mon[41619]: Saving service mgr spec with placement smithi072=y;smithi173=x;count:2 2024-03-26T01:57:27.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:27 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:27.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:27 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:27.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:27 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:27.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:27 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:27.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:27 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:27.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:27 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:57:27.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:27 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T01:57:27.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:27 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:57:27.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:27 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:27.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:27 smithi072 ceph-mon[41619]: Deploying daemon mgr.x on smithi173 2024-03-26T01:57:27.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:27 smithi072 ceph-mon[41619]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:27.496 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-03-26T01:57:27.496 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-03-26T01:57:27.496 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000227174 s, 2.3 MB/s 2024-03-26T01:57:27.497 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T01:57:27.553 DEBUG:teuthology.orchestra.run.smithi072:> stat /dev/vg_nvme/lv_4 2024-03-26T01:57:27.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:27 smithi173 ceph-mon[33839]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi072=y;smithi173=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:27.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:27 smithi173 ceph-mon[33839]: Saving service mgr spec with placement smithi072=y;smithi173=x;count:2 2024-03-26T01:57:27.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:27 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:27.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:27 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:27.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:27 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:27.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:27 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:27.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:27 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:27.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:27 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:57:27.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:27 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T01:57:27.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:27 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:57:27.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:27 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:27.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:27 smithi173 ceph-mon[33839]: Deploying daemon mgr.x on smithi173 2024-03-26T01:57:27.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:27 smithi173 ceph-mon[33839]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:27.608 INFO:teuthology.orchestra.run.smithi072.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T01:57:27.608 INFO:teuthology.orchestra.run.smithi072.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:57:27.608 INFO:teuthology.orchestra.run.smithi072.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-26T01:57:27.608 INFO:teuthology.orchestra.run.smithi072.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:57:27.608 INFO:teuthology.orchestra.run.smithi072.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:57:27.608 INFO:teuthology.orchestra.run.smithi072.stdout:Access: 2024-03-26 01:57:11.491310987 +0000 2024-03-26T01:57:27.608 INFO:teuthology.orchestra.run.smithi072.stdout:Modify: 2024-03-26 01:51:03.725490211 +0000 2024-03-26T01:57:27.608 INFO:teuthology.orchestra.run.smithi072.stdout:Change: 2024-03-26 01:51:03.726490189 +0000 2024-03-26T01:57:27.608 INFO:teuthology.orchestra.run.smithi072.stdout: Birth: 2024-03-26 01:51:03.725490211 +0000 2024-03-26T01:57:27.609 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T01:57:27.674 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records in 2024-03-26T01:57:27.674 INFO:teuthology.orchestra.run.smithi072.stderr:1+0 records out 2024-03-26T01:57:27.674 INFO:teuthology.orchestra.run.smithi072.stderr:512 bytes copied, 0.000188326 s, 2.7 MB/s 2024-03-26T01:57:27.675 DEBUG:teuthology.orchestra.run.smithi072:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T01:57:27.730 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-26T01:57:27.731 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T01:57:27.746 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T01:57:27.747 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_1 2024-03-26T01:57:27.802 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T01:57:27.802 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:57:27.802 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-26T01:57:27.802 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:57:27.802 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:57:27.802 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-26 01:57:02.442403168 +0000 2024-03-26T01:57:27.803 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-26 01:51:00.817547419 +0000 2024-03-26T01:57:27.803 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-26 01:51:00.817547419 +0000 2024-03-26T01:57:27.803 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-26 01:51:00.817547419 +0000 2024-03-26T01:57:27.803 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T01:57:27.868 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-26T01:57:27.868 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-26T01:57:27.868 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000185259 s, 2.8 MB/s 2024-03-26T01:57:27.869 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T01:57:27.925 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_2 2024-03-26T01:57:27.980 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T01:57:27.980 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:57:27.980 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-26T01:57:27.980 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:57:27.980 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:57:27.980 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-26 01:57:02.442403168 +0000 2024-03-26T01:57:27.981 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-26 01:51:01.167539762 +0000 2024-03-26T01:57:27.981 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-26 01:51:01.167539762 +0000 2024-03-26T01:57:27.981 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-26 01:51:01.167539762 +0000 2024-03-26T01:57:27.981 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T01:57:28.045 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-26T01:57:28.046 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-26T01:57:28.046 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000206666 s, 2.5 MB/s 2024-03-26T01:57:28.047 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T01:57:28.103 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_3 2024-03-26T01:57:28.158 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T01:57:28.158 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:57:28.158 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-26T01:57:28.158 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:57:28.158 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:57:28.159 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-26 01:57:02.442403168 +0000 2024-03-26T01:57:28.159 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-26 01:51:01.510532257 +0000 2024-03-26T01:57:28.159 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-26 01:51:01.510532257 +0000 2024-03-26T01:57:28.159 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-26 01:51:01.510532257 +0000 2024-03-26T01:57:28.159 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T01:57:28.338 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-26T01:57:28.338 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-26T01:57:28.338 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000197829 s, 2.6 MB/s 2024-03-26T01:57:28.339 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T01:57:28.355 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_4 2024-03-26T01:57:28.414 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T01:57:28.415 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T01:57:28.415 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-26T01:57:28.415 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T01:57:28.415 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T01:57:28.415 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-03-26 01:57:28.295835886 +0000 2024-03-26T01:57:28.415 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-03-26 01:51:01.860524599 +0000 2024-03-26T01:57:28.415 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-03-26 01:51:01.860524599 +0000 2024-03-26T01:57:28.415 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-03-26 01:51:01.860524599 +0000 2024-03-26T01:57:28.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T01:57:28.532 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-03-26T01:57:28.532 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-03-26T01:57:28.532 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.00018923 s, 2.7 MB/s 2024-03-26T01:57:28.534 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T01:57:28.549 INFO:tasks.cephadm:Deploying osd.0 on smithi072 with /dev/vg_nvme/lv_4... 2024-03-26T01:57:28.549 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T01:57:28.744 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:28 smithi173 systemd[1]: Starting Ceph mgr.x for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T01:57:28.789 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:57:29.062 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:28 smithi173 podman[35937]: 2024-03-26 01:57:28.744687117 +0000 UTC m=+0.097817139 container create 5cdf7c3ec8fe6efe8fd0532504e3c0e951628548717808f05b8e7051495aab73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2) 2024-03-26T01:57:29.063 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:28 smithi173 podman[35937]: 2024-03-26 01:57:28.664622423 +0000 UTC m=+0.017752444 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T01:57:29.063 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:28 smithi173 podman[35937]: 2024-03-26 01:57:28.955557179 +0000 UTC m=+0.308687201 container init 5cdf7c3ec8fe6efe8fd0532504e3c0e951628548717808f05b8e7051495aab73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T01:57:29.063 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:28 smithi173 podman[35937]: 2024-03-26 01:57:28.961394814 +0000 UTC m=+0.314524836 container start 5cdf7c3ec8fe6efe8fd0532504e3c0e951628548717808f05b8e7051495aab73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-26T01:57:29.063 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:28 smithi173 ceph-mgr[35973]: -- 172.21.15.173:0/2009338143 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56080498c1a0 con 0x560804962800 2024-03-26T01:57:29.526 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:29 smithi173 bash[35937]: 5cdf7c3ec8fe6efe8fd0532504e3c0e951628548717808f05b8e7051495aab73 2024-03-26T01:57:29.527 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:29 smithi173 systemd[1]: Started Ceph mgr.x for c86fbdd4-eb13-11ee-95d0-87774f69a715. 2024-03-26T01:57:29.812 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:29 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T01:57:29.526+0000 7fd4d6ca0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T01:57:29.812 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:29 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T01:57:29.621+0000 7fd4d6ca0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T01:57:30.260 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:30 smithi072 ceph-mon[30177]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:30.260 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:30 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:30.261 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:30 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:30.261 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:30 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:30.261 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:30 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:30.261 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:30 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:30.261 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:30 smithi072 ceph-mon[41619]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:30.261 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:30 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:30.261 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:30 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:30.261 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:30 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:30.261 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:30 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:30.261 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:30 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:30.385 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:30 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T01:57:30.293+0000 7fd4d6ca0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T01:57:30.386 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:30 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T01:57:30.385+0000 7fd4d6ca0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T01:57:30.386 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:30 smithi173 ceph-mon[33839]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:30.386 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:30 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:30.386 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:30 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:30.386 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:30 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:30.386 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:30 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:30.386 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:30 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:30.716 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:30 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T01:57:30.525+0000 7fd4d6ca0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T01:57:30.716 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:30 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T01:57:30.629+0000 7fd4d6ca0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T01:57:30.716 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:30 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T01:57:30.715+0000 7fd4d6ca0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T01:57:30.733 INFO:teuthology.orchestra.run.smithi072.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T01:57:30.733 INFO:teuthology.orchestra.run.smithi072.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T01:57:30.733 INFO:teuthology.orchestra.run.smithi072.stderr: stderr: 10+0 records in 2024-03-26T01:57:30.733 INFO:teuthology.orchestra.run.smithi072.stderr:10+0 records out 2024-03-26T01:57:30.733 INFO:teuthology.orchestra.run.smithi072.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115789 s, 906 MB/s 2024-03-26T01:57:30.733 INFO:teuthology.orchestra.run.smithi072.stderr:--> Zapping successful for: 2024-03-26T01:57:31.268 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:30 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T01:57:30.976+0000 7fd4d6ca0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T01:57:31.268 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:31 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T01:57:31.086+0000 7fd4d6ca0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T01:57:31.268 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:31 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T01:57:31.174+0000 7fd4d6ca0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T01:57:31.269 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:31 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T01:57:31.268+0000 7fd4d6ca0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T01:57:31.392 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi072:vg_nvme/lv_4 2024-03-26T01:57:31.562 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:31 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T01:57:31.458+0000 7fd4d6ca0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T01:57:31.886 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:31 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:31.886 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:31 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:31.887 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:31 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:31.887 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:31 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:31.887 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:31 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:31.887 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:31 smithi072 ceph-mon[41619]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T01:57:31.887 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:31 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:57:31.887 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:31 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:57:31.887 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:31 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:31.887 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:31 smithi072 ceph-mon[41619]: Reconfiguring daemon mgr.y on smithi072 2024-03-26T01:57:31.887 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:31 smithi072 ceph-mon[41619]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:31.887 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:31 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:31.887 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:31 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:31.888 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:31 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:31.888 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:31 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:31.888 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:31 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:31.888 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:31 smithi072 ceph-mon[30177]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T01:57:31.888 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:31 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:57:31.888 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:31 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:57:31.888 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:31 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:31.888 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:31 smithi072 ceph-mon[30177]: Reconfiguring daemon mgr.y on smithi072 2024-03-26T01:57:31.888 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:31 smithi072 ceph-mon[30177]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:32.062 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:31 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:31 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:31 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:31 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:31 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:31 smithi173 ceph-mon[33839]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T01:57:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:31 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T01:57:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:31 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T01:57:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:31 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:31 smithi173 ceph-mon[33839]: Reconfiguring daemon mgr.y on smithi072 2024-03-26T01:57:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:31 smithi173 ceph-mon[33839]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:32.087 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:57:32.812 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:32 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T01:57:32.426+0000 7fd4d6ca0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T01:57:32.812 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:32 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T01:57:32.556+0000 7fd4d6ca0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T01:57:33.143 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:32 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T01:57:32.892+0000 7fd4d6ca0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T01:57:33.144 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:32 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T01:57:32.982+0000 7fd4d6ca0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T01:57:33.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:33 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:33.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:33 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:33.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:33 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:33.563 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:33 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T01:57:33.176+0000 7fd4d6ca0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T01:57:33.563 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:33 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T01:57:33.358+0000 7fd4d6ca0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T01:57:33.898 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:33 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T01:57:33.640+0000 7fd4d6ca0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T01:57:33.898 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:33 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T01:57:33.897+0000 7fd4d6ca0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T01:57:34.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.030+0000 7f35dd061700 1 -- 172.21.15.72:0/2835366956 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f35d8105460 msgr2=0x7f35d8107850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:34.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.030+0000 7f35dd061700 1 --2- 172.21.15.72:0/2835366956 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f35d8105460 0x7f35d8107850 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f35cc0098f0 tx=0x7f35cc009c00 comp rx=0 tx=0).stop 2024-03-26T01:57:34.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.030+0000 7f35dd061700 1 -- 172.21.15.72:0/2835366956 shutdown_connections 2024-03-26T01:57:34.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.030+0000 7f35dd061700 1 --2- 172.21.15.72:0/2835366956 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35d8107d90 0x7f35d810a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:34.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.030+0000 7f35dd061700 1 --2- 172.21.15.72:0/2835366956 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35d8102b30 0x7f35d8104f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:34.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.030+0000 7f35dd061700 1 --2- 172.21.15.72:0/2835366956 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f35d8105460 0x7f35d8107850 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:34.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.030+0000 7f35dd061700 1 -- 172.21.15.72:0/2835366956 >> 172.21.15.72:0/2835366956 conn(0x7f35d80fc540 msgr2=0x7f35d80fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:57:34.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.031+0000 7f35dd061700 1 -- 172.21.15.72:0/2835366956 shutdown_connections 2024-03-26T01:57:34.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.031+0000 7f35dd061700 1 -- 172.21.15.72:0/2835366956 wait complete. 2024-03-26T01:57:34.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.031+0000 7f35dd061700 1 Processor -- start 2024-03-26T01:57:34.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.031+0000 7f35dd061700 1 -- start start 2024-03-26T01:57:34.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.032+0000 7f35dd061700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f35d8102b30 0x7f35d8102440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:57:34.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.032+0000 7f35dd061700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35d8105460 0x7f35d8100ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:57:34.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.032+0000 7f35dd061700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35d8107d90 0x7f35d8101020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:57:34.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.032+0000 7f35dd061700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d81017c0 con 0x7f35d8107d90 2024-03-26T01:57:34.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.032+0000 7f35dd061700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d8101930 con 0x7f35d8105460 2024-03-26T01:57:34.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.032+0000 7f35dd061700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35d81a1880 con 0x7f35d8102b30 2024-03-26T01:57:34.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.032+0000 7f35d759e700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35d8107d90 0x7f35d8101020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:57:34.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.032+0000 7f35d6d9d700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f35d8102b30 0x7f35d8102440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:57:34.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.032+0000 7f35d659c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35d8105460 0x7f35d8100ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:57:34.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.032+0000 7f35d759e700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35d8107d90 0x7f35d8101020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:49152/0 (socket says 172.21.15.72:49152) 2024-03-26T01:57:34.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.032+0000 7f35d759e700 1 -- 172.21.15.72:0/1661030034 learned_addr learned my addr 172.21.15.72:0/1661030034 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:57:34.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.032+0000 7f35d759e700 1 -- 172.21.15.72:0/1661030034 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f35d8102b30 msgr2=0x7f35d8102440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:34.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.032+0000 7f35d759e700 1 --2- 172.21.15.72:0/1661030034 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f35d8102b30 0x7f35d8102440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:34.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.032+0000 7f35d759e700 1 -- 172.21.15.72:0/1661030034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35d8105460 msgr2=0x7f35d8100ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:34.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.032+0000 7f35d759e700 1 --2- 172.21.15.72:0/1661030034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35d8105460 0x7f35d8100ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:34.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.032+0000 7f35d759e700 1 -- 172.21.15.72:0/1661030034 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c80095f0 con 0x7f35d8107d90 2024-03-26T01:57:34.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.032+0000 7f35d759e700 1 --2- 172.21.15.72:0/1661030034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35d8107d90 0x7f35d8101020 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f35d00094f0 tx=0x7f35d000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:57:34.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.033+0000 7f35bffff700 1 -- 172.21.15.72:0/1661030034 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d0004db0 con 0x7f35d8107d90 2024-03-26T01:57:34.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.033+0000 7f35bffff700 1 -- 172.21.15.72:0/1661030034 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f35d000f040 con 0x7f35d8107d90 2024-03-26T01:57:34.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.033+0000 7f35bffff700 1 -- 172.21.15.72:0/1661030034 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d0005940 con 0x7f35d8107d90 2024-03-26T01:57:34.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.033+0000 7f35dd061700 1 -- 172.21.15.72:0/1661030034 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35cc009520 con 0x7f35d8107d90 2024-03-26T01:57:34.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.033+0000 7f35dd061700 1 -- 172.21.15.72:0/1661030034 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35d81a1e80 con 0x7f35d8107d90 2024-03-26T01:57:34.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.034+0000 7f35bffff700 1 -- 172.21.15.72:0/1661030034 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 13) v1 ==== 45949+0+0 (secure 0 0 0) 0x7f35d0015400 con 0x7f35d8107d90 2024-03-26T01:57:34.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.034+0000 7f35bffff700 1 --2- 172.21.15.72:0/1661030034 >> v2:172.21.15.72:6800/2923394687 conn(0x7f35c0039180 0x7f35c003b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:57:34.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.034+0000 7f35bffff700 1 -- 172.21.15.72:0/1661030034 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (secure 0 0 0) 0x7f35d004b3c0 con 0x7f35d8107d90 2024-03-26T01:57:34.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.034+0000 7f35d6d9d700 1 --2- 172.21.15.72:0/1661030034 >> v2:172.21.15.72:6800/2923394687 conn(0x7f35c0039180 0x7f35c003b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:57:34.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.034+0000 7f35dd061700 1 -- 172.21.15.72:0/1661030034 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35d80fe0c0 con 0x7f35d8107d90 2024-03-26T01:57:34.036 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.034+0000 7f35d6d9d700 1 --2- 172.21.15.72:0/1661030034 >> v2:172.21.15.72:6800/2923394687 conn(0x7f35c0039180 0x7f35c003b640 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f35c8005fd0 tx=0x7f35c8009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:57:34.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.042+0000 7f35bffff700 1 -- 172.21.15.72:0/1661030034 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f35d0014950 con 0x7f35d8107d90 2024-03-26T01:57:34.261 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:34.260+0000 7f35dd061700 1 -- 172.21.15.72:0/1661030034 --> v2:172.21.15.72:6800/2923394687 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f35d810d2c0 con 0x7f35c0039180 2024-03-26T01:57:34.312 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:34 smithi173 ceph-mon[33839]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:34.312 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:34 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:34.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:34 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:34.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:34 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:34.313 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:34 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T01:57:34.132+0000 7fd4d6ca0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T01:57:34.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:34 smithi072 ceph-mon[30177]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:34.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:34 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:34.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:34 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:34.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:34 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:34.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:34 smithi072 ceph-mon[41619]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:34.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:34 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:34.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:34 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:34.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:34 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:35.312 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:34 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T01:57:34.934+0000 7fd4d6ca0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T01:57:35.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:35 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:35.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:35 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:35.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:35 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:35.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:35 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:57:35.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:35 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:57:35.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:35 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:35.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:35 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:35.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:35 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:35.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:35 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:35.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:35 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:57:35.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:35 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:57:35.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:35 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:35.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:35 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:35.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:35 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:35.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:35 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:35.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:35 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:57:35.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:35 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:57:35.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:35 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:35.812 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:35 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T01:57:35.501+0000 7fd4d6ca0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T01:57:35.812 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:35 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T01:57:35.597+0000 7fd4d6ca0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T01:57:35.812 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 01:57:35 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T01:57:35.811+0000 7fd4d6ca0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T01:57:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:36 smithi072 ceph-mon[30177]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:36 smithi072 ceph-mon[30177]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:36 smithi072 ceph-mon[30177]: Standby manager daemon x started 2024-03-26T01:57:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:36 smithi072 ceph-mon[30177]: from='mgr.? 172.21.15.173:0/3545597507' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T01:57:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:36 smithi072 ceph-mon[30177]: from='mgr.? 172.21.15.173:0/3545597507' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:57:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:36 smithi072 ceph-mon[30177]: from='mgr.? 172.21.15.173:0/3545597507' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T01:57:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:36 smithi072 ceph-mon[30177]: from='mgr.? 172.21.15.173:0/3545597507' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:57:36.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:36 smithi072 ceph-mon[41619]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:36.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:36 smithi072 ceph-mon[41619]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:36.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:36 smithi072 ceph-mon[41619]: Standby manager daemon x started 2024-03-26T01:57:36.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:36 smithi072 ceph-mon[41619]: from='mgr.? 172.21.15.173:0/3545597507' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T01:57:36.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:36 smithi072 ceph-mon[41619]: from='mgr.? 172.21.15.173:0/3545597507' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:57:36.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:36 smithi072 ceph-mon[41619]: from='mgr.? 172.21.15.173:0/3545597507' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T01:57:36.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:36 smithi072 ceph-mon[41619]: from='mgr.? 172.21.15.173:0/3545597507' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:57:36.254 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:36.254+0000 7f35bffff700 1 -- 172.21.15.72:0/1661030034 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mgrmap(e 14) v1 ==== 91307+0+0 (secure 0 0 0) 0x7f35d0015700 con 0x7f35d8107d90 2024-03-26T01:57:36.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:36 smithi173 ceph-mon[33839]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:57:36.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:36 smithi173 ceph-mon[33839]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:36.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:36 smithi173 ceph-mon[33839]: Standby manager daemon x started 2024-03-26T01:57:36.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:36 smithi173 ceph-mon[33839]: from='mgr.? 172.21.15.173:0/3545597507' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T01:57:36.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:36 smithi173 ceph-mon[33839]: from='mgr.? 172.21.15.173:0/3545597507' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T01:57:36.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:36 smithi173 ceph-mon[33839]: from='mgr.? 172.21.15.173:0/3545597507' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T01:57:36.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:36 smithi173 ceph-mon[33839]: from='mgr.? 172.21.15.173:0/3545597507' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T01:57:37.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:37 smithi173 ceph-mon[33839]: mgrmap e14: y(active, since 69s), standbys: x 2024-03-26T01:57:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:37 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T01:57:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:37 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/765167042' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f311af9-e18d-42fa-9032-8fa8081d2c4f"}]: dispatch 2024-03-26T01:57:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:37 smithi173 ceph-mon[33839]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f311af9-e18d-42fa-9032-8fa8081d2c4f"}]: dispatch 2024-03-26T01:57:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:37 smithi173 ceph-mon[33839]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0f311af9-e18d-42fa-9032-8fa8081d2c4f"}]': finished 2024-03-26T01:57:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:37 smithi173 ceph-mon[33839]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T01:57:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:37 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:37 smithi173 ceph-mon[33839]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:37.718 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:37 smithi072 ceph-mon[30177]: mgrmap e14: y(active, since 69s), standbys: x 2024-03-26T01:57:37.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:37 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T01:57:37.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:37 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/765167042' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f311af9-e18d-42fa-9032-8fa8081d2c4f"}]: dispatch 2024-03-26T01:57:37.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:37 smithi072 ceph-mon[30177]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f311af9-e18d-42fa-9032-8fa8081d2c4f"}]: dispatch 2024-03-26T01:57:37.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:37 smithi072 ceph-mon[30177]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0f311af9-e18d-42fa-9032-8fa8081d2c4f"}]': finished 2024-03-26T01:57:37.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:37 smithi072 ceph-mon[30177]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T01:57:37.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:37 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:37.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:37 smithi072 ceph-mon[30177]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:37.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:37 smithi072 ceph-mon[41619]: mgrmap e14: y(active, since 69s), standbys: x 2024-03-26T01:57:37.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:37 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T01:57:37.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:37 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/765167042' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f311af9-e18d-42fa-9032-8fa8081d2c4f"}]: dispatch 2024-03-26T01:57:37.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:37 smithi072 ceph-mon[41619]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0f311af9-e18d-42fa-9032-8fa8081d2c4f"}]: dispatch 2024-03-26T01:57:37.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:37 smithi072 ceph-mon[41619]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0f311af9-e18d-42fa-9032-8fa8081d2c4f"}]': finished 2024-03-26T01:57:37.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:37 smithi072 ceph-mon[41619]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T01:57:37.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:37 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:37.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:37 smithi072 ceph-mon[41619]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:38.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:38 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1909946275' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:57:38.619 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:38 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1909946275' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:57:38.620 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:38 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1909946275' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:57:39.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:39 smithi173 ceph-mon[33839]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:39.718 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:39 smithi072 ceph-mon[30177]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:39.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:39 smithi072 ceph-mon[41619]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:42.210 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:42 smithi072 ceph-mon[30177]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:42.211 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:42 smithi072 ceph-mon[41619]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:42 smithi173 ceph-mon[33839]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:44.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:44 smithi072 ceph-mon[30177]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:44.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:44 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T01:57:44.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:44 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:44.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:44 smithi072 ceph-mon[41619]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:44.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:44 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T01:57:44.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:44 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:44.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:44 smithi173 ceph-mon[33839]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:44.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:44 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T01:57:44.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:44 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:45.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:45 smithi072 ceph-mon[30177]: Deploying daemon osd.0 on smithi072 2024-03-26T01:57:45.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:45 smithi072 ceph-mon[41619]: Deploying daemon osd.0 on smithi072 2024-03-26T01:57:45.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:45 smithi173 ceph-mon[33839]: Deploying daemon osd.0 on smithi072 2024-03-26T01:57:46.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:46 smithi072 ceph-mon[30177]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:46.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:46 smithi072 ceph-mon[41619]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:46.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:46 smithi173 ceph-mon[33839]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:48.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:48 smithi072 ceph-mon[30177]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:48.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:48 smithi072 ceph-mon[41619]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:48.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:48 smithi173 ceph-mon[33839]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:49.968 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:49 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:49.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:49 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:49.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:49 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:49.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:49 smithi072 ceph-mon[30177]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:49.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:49 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:49.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:49 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:49.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:49 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:49.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:49 smithi072 ceph-mon[41619]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:50.062 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:49 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:50.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:49 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:50.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:49 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:50.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:49 smithi173 ceph-mon[33839]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:52.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:52 smithi072 ceph-mon[30177]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:52.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:52 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:52.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:52 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:52.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:52 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:52.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:52 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:52.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:52 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:52.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:52 smithi072 ceph-mon[30177]: from='osd.0 v2:172.21.15.72:6801/1438405769' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T01:57:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:52 smithi072 ceph-mon[41619]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:52 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:52 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:52 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:52 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:52 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:52 smithi072 ceph-mon[41619]: from='osd.0 v2:172.21.15.72:6801/1438405769' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T01:57:52.512 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:52.512+0000 7f35bffff700 1 -- 172.21.15.72:0/1661030034 <== mgr.14152 v2:172.21.15.72:6800/2923394687 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f35d810d2c0 con 0x7f35c0039180 2024-03-26T01:57:52.512 INFO:teuthology.orchestra.run.smithi072.stdout:Created osd(s) 0 on host 'smithi072' 2024-03-26T01:57:52.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:52.514+0000 7f35dd061700 1 -- 172.21.15.72:0/1661030034 >> v2:172.21.15.72:6800/2923394687 conn(0x7f35c0039180 msgr2=0x7f35c003b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:52.514 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:52.514+0000 7f35dd061700 1 --2- 172.21.15.72:0/1661030034 >> v2:172.21.15.72:6800/2923394687 conn(0x7f35c0039180 0x7f35c003b640 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f35c8005fd0 tx=0x7f35c8009310 comp rx=0 tx=0).stop 2024-03-26T01:57:52.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:52.514+0000 7f35dd061700 1 -- 172.21.15.72:0/1661030034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35d8107d90 msgr2=0x7f35d8101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:52.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:52.514+0000 7f35dd061700 1 --2- 172.21.15.72:0/1661030034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35d8107d90 0x7f35d8101020 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f35d00094f0 tx=0x7f35d000d3b0 comp rx=0 tx=0).stop 2024-03-26T01:57:52.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:52.514+0000 7f35dd061700 1 -- 172.21.15.72:0/1661030034 shutdown_connections 2024-03-26T01:57:52.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:52.514+0000 7f35dd061700 1 --2- 172.21.15.72:0/1661030034 >> v2:172.21.15.72:6800/2923394687 conn(0x7f35c0039180 0x7f35c003b640 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:52.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:52.514+0000 7f35dd061700 1 --2- 172.21.15.72:0/1661030034 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f35d8107d90 0x7f35d8101020 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:52.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:52.514+0000 7f35dd061700 1 --2- 172.21.15.72:0/1661030034 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f35d8102b30 0x7f35d8102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:52.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:52.514+0000 7f35dd061700 1 --2- 172.21.15.72:0/1661030034 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f35d8105460 0x7f35d8100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:52.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:52.514+0000 7f35dd061700 1 -- 172.21.15.72:0/1661030034 >> 172.21.15.72:0/1661030034 conn(0x7f35d80fc540 msgr2=0x7f35d81037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:57:52.515 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:52.514+0000 7f35dd061700 1 -- 172.21.15.72:0/1661030034 shutdown_connections 2024-03-26T01:57:52.516 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:52.514+0000 7f35dd061700 1 -- 172.21.15.72:0/1661030034 wait complete. 2024-03-26T01:57:52.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:52 smithi173 ceph-mon[33839]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:52.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:52 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:52.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:52 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:52.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:52 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:52.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:52 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:52.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:52 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:52.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:52 smithi173 ceph-mon[33839]: from='osd.0 v2:172.21.15.72:6801/1438405769' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T01:57:53.114 DEBUG:teuthology.orchestra.run.smithi072:osd.0> sudo journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.0.service 2024-03-26T01:57:53.116 INFO:tasks.cephadm:Deploying osd.1 on smithi072 with /dev/vg_nvme/lv_3... 2024-03-26T01:57:53.117 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T01:57:53.672 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:53 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:53.672 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:53 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:53.672 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:53 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:53.672 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:53 smithi072 ceph-mon[41619]: from='osd.0 v2:172.21.15.72:6801/1438405769' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T01:57:53.672 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:53 smithi072 ceph-mon[41619]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T01:57:53.672 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:53 smithi072 ceph-mon[41619]: from='osd.0 v2:172.21.15.72:6801/1438405769' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-26T01:57:53.672 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:53 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:53.672 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:53 smithi072 ceph-mon[41619]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:53.673 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:53 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:53.673 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:53 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:53.673 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:53 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:53.673 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:53 smithi072 ceph-mon[30177]: from='osd.0 v2:172.21.15.72:6801/1438405769' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T01:57:53.673 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:53 smithi072 ceph-mon[30177]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T01:57:53.673 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:53 smithi072 ceph-mon[30177]: from='osd.0 v2:172.21.15.72:6801/1438405769' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-26T01:57:53.673 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:53 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:53.674 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:53 smithi072 ceph-mon[30177]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:53.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:53 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:53.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:53 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:53.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:53 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:57:53.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:53 smithi173 ceph-mon[33839]: from='osd.0 v2:172.21.15.72:6801/1438405769' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T01:57:53.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:53 smithi173 ceph-mon[33839]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T01:57:53.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:53 smithi173 ceph-mon[33839]: from='osd.0 v2:172.21.15.72:6801/1438405769' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-26T01:57:53.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:53 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:53.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:53 smithi173 ceph-mon[33839]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:53.926 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:57:54.297 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 26 01:57:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-0[46061]: 2024-03-26T01:57:54.102+0000 7f18342fd700 -1 osd.0 0 waiting for initial osdmap 2024-03-26T01:57:54.297 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 26 01:57:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-0[46061]: 2024-03-26T01:57:54.107+0000 7f182b8e7700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T01:57:54.830 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:54 smithi072 ceph-mon[30177]: from='osd.0 v2:172.21.15.72:6801/1438405769' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-03-26T01:57:54.830 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:54 smithi072 ceph-mon[30177]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T01:57:54.830 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:54 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:54.830 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:54 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:54.831 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:54 smithi072 ceph-mon[30177]: from='osd.0 v2:172.21.15.72:6801/1438405769' entity='osd.0' 2024-03-26T01:57:54.831 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:54 smithi072 ceph-mon[41619]: from='osd.0 v2:172.21.15.72:6801/1438405769' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-03-26T01:57:54.831 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:54 smithi072 ceph-mon[41619]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T01:57:54.831 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:54 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:54.831 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:54 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:54.831 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:54 smithi072 ceph-mon[41619]: from='osd.0 v2:172.21.15.72:6801/1438405769' entity='osd.0' 2024-03-26T01:57:55.062 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:54 smithi173 ceph-mon[33839]: from='osd.0 v2:172.21.15.72:6801/1438405769' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-03-26T01:57:55.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:54 smithi173 ceph-mon[33839]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T01:57:55.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:54 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:55.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:54 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:55.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:54 smithi173 ceph-mon[33839]: from='osd.0 v2:172.21.15.72:6801/1438405769' entity='osd.0' 2024-03-26T01:57:55.828 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:55 smithi072 ceph-mon[30177]: purged_snaps scrub starts 2024-03-26T01:57:55.828 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:55 smithi072 ceph-mon[30177]: purged_snaps scrub ok 2024-03-26T01:57:55.828 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:55 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:55.828 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:55 smithi072 ceph-mon[30177]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:55.828 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:55 smithi072 ceph-mon[30177]: osd.0 v2:172.21.15.72:6801/1438405769 boot 2024-03-26T01:57:55.828 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:55 smithi072 ceph-mon[30177]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T01:57:55.829 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:55 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:55.829 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:55 smithi072 ceph-mon[41619]: purged_snaps scrub starts 2024-03-26T01:57:55.829 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:55 smithi072 ceph-mon[41619]: purged_snaps scrub ok 2024-03-26T01:57:55.829 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:55 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:55.829 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:55 smithi072 ceph-mon[41619]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:55.829 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:55 smithi072 ceph-mon[41619]: osd.0 v2:172.21.15.72:6801/1438405769 boot 2024-03-26T01:57:55.829 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:55 smithi072 ceph-mon[41619]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T01:57:55.829 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:55 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:56.062 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:55 smithi173 ceph-mon[33839]: purged_snaps scrub starts 2024-03-26T01:57:56.062 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:55 smithi173 ceph-mon[33839]: purged_snaps scrub ok 2024-03-26T01:57:56.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:55 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:56.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:55 smithi173 ceph-mon[33839]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T01:57:56.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:55 smithi173 ceph-mon[33839]: osd.0 v2:172.21.15.72:6801/1438405769 boot 2024-03-26T01:57:56.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:55 smithi173 ceph-mon[33839]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T01:57:56.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:55 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T01:57:57.336 INFO:teuthology.orchestra.run.smithi072.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T01:57:57.336 INFO:teuthology.orchestra.run.smithi072.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T01:57:57.337 INFO:teuthology.orchestra.run.smithi072.stderr: stderr: 10+0 records in 2024-03-26T01:57:57.337 INFO:teuthology.orchestra.run.smithi072.stderr:10+0 records out 2024-03-26T01:57:57.337 INFO:teuthology.orchestra.run.smithi072.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126937 s, 826 MB/s 2024-03-26T01:57:57.337 INFO:teuthology.orchestra.run.smithi072.stderr:--> Zapping successful for: 2024-03-26T01:57:57.674 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:57 smithi072 ceph-mon[30177]: Detected new or changed devices on smithi072 2024-03-26T01:57:57.674 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:57 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:57.674 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:57 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:57.674 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:57 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:57:57.674 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:57 smithi072 ceph-mon[30177]: Adjusting osd_memory_target on smithi072 to 16064M 2024-03-26T01:57:57.674 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:57 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:57.674 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:57 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:57.674 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:57 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:57.675 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:57 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:57.675 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:57 smithi072 ceph-mon[30177]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T01:57:57.675 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:57:57 smithi072 ceph-mon[30177]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:57.675 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:57 smithi072 ceph-mon[41619]: Detected new or changed devices on smithi072 2024-03-26T01:57:57.675 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:57 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:57.675 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:57 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:57.675 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:57 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:57:57.675 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:57 smithi072 ceph-mon[41619]: Adjusting osd_memory_target on smithi072 to 16064M 2024-03-26T01:57:57.675 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:57 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:57.675 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:57 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:57.675 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:57 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:57.676 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:57 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:57.676 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:57 smithi072 ceph-mon[41619]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T01:57:57.676 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:57:57 smithi072 ceph-mon[41619]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:57.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:57 smithi173 ceph-mon[33839]: Detected new or changed devices on smithi072 2024-03-26T01:57:57.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:57 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:57.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:57 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:57.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:57 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:57:57.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:57 smithi173 ceph-mon[33839]: Adjusting osd_memory_target on smithi072 to 16064M 2024-03-26T01:57:57.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:57 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:57.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:57 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:57:57.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:57 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:57:57.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:57 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:57:57.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:57 smithi173 ceph-mon[33839]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T01:57:57.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:57:57 smithi173 ceph-mon[33839]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:57:57.990 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi072:vg_nvme/lv_3 2024-03-26T01:57:58.226 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:57:59.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.860+0000 7fe569622700 1 -- 172.21.15.72:0/55032160 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5641021c0 msgr2=0x7fe564102620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:59.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.860+0000 7fe569622700 1 --2- 172.21.15.72:0/55032160 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5641021c0 0x7fe564102620 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe5580099c0 tx=0x7fe558009cd0 comp rx=0 tx=0).stop 2024-03-26T01:57:59.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.860+0000 7fe569622700 1 -- 172.21.15.72:0/55032160 shutdown_connections 2024-03-26T01:57:59.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.860+0000 7fe569622700 1 --2- 172.21.15.72:0/55032160 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe564102b60 0x7fe564109410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:59.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.860+0000 7fe569622700 1 --2- 172.21.15.72:0/55032160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe564100fc0 0x7fe5641013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:59.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.860+0000 7fe569622700 1 --2- 172.21.15.72:0/55032160 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe5641021c0 0x7fe564102620 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:59.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.860+0000 7fe569622700 1 -- 172.21.15.72:0/55032160 >> 172.21.15.72:0/55032160 conn(0x7fe5640fc580 msgr2=0x7fe5640fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:57:59.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.860+0000 7fe569622700 1 -- 172.21.15.72:0/55032160 shutdown_connections 2024-03-26T01:57:59.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.860+0000 7fe569622700 1 -- 172.21.15.72:0/55032160 wait complete. 2024-03-26T01:57:59.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.861+0000 7fe569622700 1 Processor -- start 2024-03-26T01:57:59.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.861+0000 7fe569622700 1 -- start start 2024-03-26T01:57:59.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.861+0000 7fe569622700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe564100fc0 0x7fe56406b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:57:59.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.861+0000 7fe569622700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5641021c0 0x7fe56419c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:57:59.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.861+0000 7fe569622700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe564102b60 0x7fe56419eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:57:59.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.861+0000 7fe569622700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56419f2d0 con 0x7fe564102b60 2024-03-26T01:57:59.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.861+0000 7fe569622700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe56419f440 con 0x7fe5641021c0 2024-03-26T01:57:59.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.861+0000 7fe569622700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe56419f600 con 0x7fe564100fc0 2024-03-26T01:57:59.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.862+0000 7fe5637fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe564102b60 0x7fe56419eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:57:59.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.862+0000 7fe562ffd700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe564100fc0 0x7fe56406b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:57:59.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.862+0000 7fe562ffd700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe564100fc0 0x7fe56406b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:51928/0 (socket says 172.21.15.72:51928) 2024-03-26T01:57:59.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.862+0000 7fe562ffd700 1 -- 172.21.15.72:0/354376543 learned_addr learned my addr 172.21.15.72:0/354376543 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:57:59.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.862+0000 7fe5627fc700 1 --2- 172.21.15.72:0/354376543 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5641021c0 0x7fe56419c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:57:59.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.862+0000 7fe5637fe700 1 -- 172.21.15.72:0/354376543 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe564100fc0 msgr2=0x7fe56406b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:59.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.862+0000 7fe5637fe700 1 --2- 172.21.15.72:0/354376543 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe564100fc0 0x7fe56406b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:59.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.862+0000 7fe5637fe700 1 -- 172.21.15.72:0/354376543 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5641021c0 msgr2=0x7fe56419c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:57:59.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.862+0000 7fe5637fe700 1 --2- 172.21.15.72:0/354376543 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5641021c0 0x7fe56419c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:57:59.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.862+0000 7fe5637fe700 1 -- 172.21.15.72:0/354376543 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55c009c40 con 0x7fe564102b60 2024-03-26T01:57:59.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.862+0000 7fe5627fc700 1 --2- 172.21.15.72:0/354376543 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5641021c0 0x7fe56419c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T01:57:59.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.862+0000 7fe5637fe700 1 --2- 172.21.15.72:0/354376543 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe564102b60 0x7fe56419eb30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00ec00 tx=0x7fe55c00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:57:59.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.862+0000 7fe54bfff700 1 -- 172.21.15.72:0/354376543 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c003a50 con 0x7fe564102b60 2024-03-26T01:57:59.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.862+0000 7fe569622700 1 -- 172.21.15.72:0/354376543 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5580095f0 con 0x7fe564102b60 2024-03-26T01:57:59.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.862+0000 7fe54bfff700 1 -- 172.21.15.72:0/354376543 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fe55c00f040 con 0x7fe564102b60 2024-03-26T01:57:59.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.862+0000 7fe54bfff700 1 -- 172.21.15.72:0/354376543 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c004290 con 0x7fe564102b60 2024-03-26T01:57:59.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.862+0000 7fe569622700 1 -- 172.21.15.72:0/354376543 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe56419fc10 con 0x7fe564102b60 2024-03-26T01:57:59.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.863+0000 7fe569622700 1 -- 172.21.15.72:0/354376543 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe564107ba0 con 0x7fe564102b60 2024-03-26T01:57:59.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.864+0000 7fe54bfff700 1 -- 172.21.15.72:0/354376543 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 14) v1 ==== 91307+0+0 (secure 0 0 0) 0x7fe55c00c680 con 0x7fe564102b60 2024-03-26T01:57:59.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.864+0000 7fe54bfff700 1 --2- 172.21.15.72:0/354376543 >> v2:172.21.15.72:6800/2923394687 conn(0x7fe54c06d900 0x7fe54c06fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:57:59.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.864+0000 7fe562ffd700 1 --2- 172.21.15.72:0/354376543 >> v2:172.21.15.72:6800/2923394687 conn(0x7fe54c06d900 0x7fe54c06fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:57:59.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.864+0000 7fe54bfff700 1 -- 172.21.15.72:0/354376543 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1541+0+0 (secure 0 0 0) 0x7fe55c08bbd0 con 0x7fe564102b60 2024-03-26T01:57:59.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.864+0000 7fe562ffd700 1 --2- 172.21.15.72:0/354376543 >> v2:172.21.15.72:6800/2923394687 conn(0x7fe54c06d900 0x7fe54c06fdc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe554005950 tx=0x7fe554009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:57:59.869 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:57:59.869+0000 7fe54bfff700 1 -- 172.21.15.72:0/354376543 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe55c056810 con 0x7fe564102b60 2024-03-26T01:58:00.074 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:00.074+0000 7fe569622700 1 -- 172.21.15.72:0/354376543 --> v2:172.21.15.72:6800/2923394687 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fe56410c270 con 0x7fe54c06d900 2024-03-26T01:58:00.468 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:00 smithi072 ceph-mon[41619]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:00.468 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:00 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:58:00.468 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:00 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:58:00.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:00 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:00.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:00 smithi072 ceph-mon[30177]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:00.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:00 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:58:00.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:00 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:58:00.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:00 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:00.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:00 smithi173 ceph-mon[33839]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:00.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:00 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:58:00.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:00 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:58:00.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:00 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:01.218 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:01 smithi072 ceph-mon[30177]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:01.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:01 smithi072 ceph-mon[41619]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:01.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:01 smithi173 ceph-mon[33839]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:02 smithi072 ceph-mon[30177]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:02.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:02 smithi072 ceph-mon[41619]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:02.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:02 smithi173 ceph-mon[33839]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3130828872' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9db9d82b-f680-418a-9192-88eb5fe854b2"}]: dispatch 2024-03-26T01:58:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:03 smithi072 ceph-mon[30177]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9db9d82b-f680-418a-9192-88eb5fe854b2"}]: dispatch 2024-03-26T01:58:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:03 smithi072 ceph-mon[30177]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9db9d82b-f680-418a-9192-88eb5fe854b2"}]': finished 2024-03-26T01:58:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:03 smithi072 ceph-mon[30177]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T01:58:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:03 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:58:03.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3130828872' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9db9d82b-f680-418a-9192-88eb5fe854b2"}]: dispatch 2024-03-26T01:58:03.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:03 smithi072 ceph-mon[41619]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9db9d82b-f680-418a-9192-88eb5fe854b2"}]: dispatch 2024-03-26T01:58:03.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:03 smithi072 ceph-mon[41619]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9db9d82b-f680-418a-9192-88eb5fe854b2"}]': finished 2024-03-26T01:58:03.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:03 smithi072 ceph-mon[41619]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T01:58:03.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:03 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:58:03.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3130828872' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9db9d82b-f680-418a-9192-88eb5fe854b2"}]: dispatch 2024-03-26T01:58:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:03 smithi173 ceph-mon[33839]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9db9d82b-f680-418a-9192-88eb5fe854b2"}]: dispatch 2024-03-26T01:58:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:03 smithi173 ceph-mon[33839]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9db9d82b-f680-418a-9192-88eb5fe854b2"}]': finished 2024-03-26T01:58:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:03 smithi173 ceph-mon[33839]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T01:58:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:03 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:58:04.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:04 smithi072 ceph-mon[30177]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:04.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:04 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2954980418' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:58:04.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:04 smithi072 ceph-mon[41619]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:04.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:04 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2954980418' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:58:04.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:04 smithi173 ceph-mon[33839]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:04.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:04 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2954980418' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:58:06.373 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:06 smithi072 ceph-mon[30177]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:06.374 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:06 smithi072 ceph-mon[41619]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:06.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:06 smithi173 ceph-mon[33839]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:08.175 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:08 smithi072 ceph-mon[30177]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:08.175 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:08 smithi072 ceph-mon[41619]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:08.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:08 smithi173 ceph-mon[33839]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:10.458 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:10 smithi072 ceph-mon[30177]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:10.458 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:10 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T01:58:10.458 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:10 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:10.458 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:10 smithi072 ceph-mon[41619]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:10.458 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:10 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T01:58:10.458 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:10 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:10.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:10 smithi173 ceph-mon[33839]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:10.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:10 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T01:58:10.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:10 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:11.340 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:11 smithi072 ceph-mon[30177]: Deploying daemon osd.1 on smithi072 2024-03-26T01:58:11.341 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:11 smithi072 ceph-mon[41619]: Deploying daemon osd.1 on smithi072 2024-03-26T01:58:11.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:11 smithi173 ceph-mon[33839]: Deploying daemon osd.1 on smithi072 2024-03-26T01:58:12.334 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:12 smithi072 ceph-mon[30177]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:12.335 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:12 smithi072 ceph-mon[41619]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:12.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:12 smithi173 ceph-mon[33839]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:14.360 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:14 smithi072 ceph-mon[30177]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:14.360 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:14 smithi072 ceph-mon[41619]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:14.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:14 smithi173 ceph-mon[33839]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:16.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:15 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:16.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:15 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:16.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:15 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:16.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:15 smithi072 ceph-mon[41619]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:15 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:15 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:15 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:15 smithi072 ceph-mon[30177]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:16.312 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:15 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:16.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:15 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:16.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:15 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:16.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:15 smithi173 ceph-mon[33839]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:18.368 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:18 smithi072 ceph-mon[41619]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:18.368 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:18 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:18.368 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:18 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:18.368 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:18 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:18.368 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:18 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:18.368 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:18 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:18.368 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:18 smithi072 ceph-mon[41619]: from='osd.1 v2:172.21.15.72:6805/2073231730' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T01:58:18.369 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:18 smithi072 ceph-mon[30177]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:18.369 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:18 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:18.369 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:18 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:18.369 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:18 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:18.369 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:18 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:18.369 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:18 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:18.369 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:18 smithi072 ceph-mon[30177]: from='osd.1 v2:172.21.15.72:6805/2073231730' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T01:58:18.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:18 smithi173 ceph-mon[33839]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:18.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:18 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:18.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:18 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:18.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:18 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:18.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:18 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:18.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:18 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:18.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:18 smithi173 ceph-mon[33839]: from='osd.1 v2:172.21.15.72:6805/2073231730' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T01:58:18.729 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:18.729+0000 7fe54bfff700 1 -- 172.21.15.72:0/354376543 <== mgr.14152 v2:172.21.15.72:6800/2923394687 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe56410c270 con 0x7fe54c06d900 2024-03-26T01:58:18.730 INFO:teuthology.orchestra.run.smithi072.stdout:Created osd(s) 1 on host 'smithi072' 2024-03-26T01:58:18.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:18.733+0000 7fe569622700 1 -- 172.21.15.72:0/354376543 >> v2:172.21.15.72:6800/2923394687 conn(0x7fe54c06d900 msgr2=0x7fe54c06fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:58:18.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:18.733+0000 7fe569622700 1 --2- 172.21.15.72:0/354376543 >> v2:172.21.15.72:6800/2923394687 conn(0x7fe54c06d900 0x7fe54c06fdc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe554005950 tx=0x7fe554009260 comp rx=0 tx=0).stop 2024-03-26T01:58:18.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:18.733+0000 7fe569622700 1 -- 172.21.15.72:0/354376543 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe564102b60 msgr2=0x7fe56419eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:58:18.733 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:18.733+0000 7fe569622700 1 --2- 172.21.15.72:0/354376543 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe564102b60 0x7fe56419eb30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00ec00 tx=0x7fe55c00ef10 comp rx=0 tx=0).stop 2024-03-26T01:58:18.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:18.733+0000 7fe569622700 1 -- 172.21.15.72:0/354376543 shutdown_connections 2024-03-26T01:58:18.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:18.733+0000 7fe569622700 1 --2- 172.21.15.72:0/354376543 >> v2:172.21.15.72:6800/2923394687 conn(0x7fe54c06d900 0x7fe54c06fdc0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:58:18.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:18.733+0000 7fe569622700 1 --2- 172.21.15.72:0/354376543 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe564102b60 0x7fe56419eb30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:58:18.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:18.733+0000 7fe569622700 1 --2- 172.21.15.72:0/354376543 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe564100fc0 0x7fe56406b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:58:18.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:18.733+0000 7fe569622700 1 --2- 172.21.15.72:0/354376543 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5641021c0 0x7fe56419c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:58:18.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:18.733+0000 7fe569622700 1 -- 172.21.15.72:0/354376543 >> 172.21.15.72:0/354376543 conn(0x7fe5640fc580 msgr2=0x7fe5640fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:58:18.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:18.733+0000 7fe569622700 1 -- 172.21.15.72:0/354376543 shutdown_connections 2024-03-26T01:58:18.734 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:18.733+0000 7fe569622700 1 -- 172.21.15.72:0/354376543 wait complete. 2024-03-26T01:58:19.363 DEBUG:teuthology.orchestra.run.smithi072:osd.1> sudo journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.1.service 2024-03-26T01:58:19.365 INFO:tasks.cephadm:Deploying osd.2 on smithi072 with /dev/vg_nvme/lv_2... 2024-03-26T01:58:19.366 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T01:58:19.938 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:19 smithi072 ceph-mon[30177]: from='osd.1 v2:172.21.15.72:6805/2073231730' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T01:58:19.938 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:19 smithi072 ceph-mon[30177]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T01:58:19.938 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:19 smithi072 ceph-mon[30177]: from='osd.1 v2:172.21.15.72:6805/2073231730' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-26T01:58:19.938 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:19 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:58:19.938 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:19 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:19.938 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:19 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:19.938 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:19 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:19.938 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:19 smithi072 ceph-mon[30177]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:19.939 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:19 smithi072 ceph-mon[41619]: from='osd.1 v2:172.21.15.72:6805/2073231730' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T01:58:19.939 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:19 smithi072 ceph-mon[41619]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T01:58:19.939 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:19 smithi072 ceph-mon[41619]: from='osd.1 v2:172.21.15.72:6805/2073231730' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-26T01:58:19.939 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:19 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:58:19.939 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:19 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:19.939 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:19 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:19.939 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:19 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:19.939 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:19 smithi072 ceph-mon[41619]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:20.062 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:19 smithi173 ceph-mon[33839]: from='osd.1 v2:172.21.15.72:6805/2073231730' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T01:58:20.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:19 smithi173 ceph-mon[33839]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T01:58:20.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:19 smithi173 ceph-mon[33839]: from='osd.1 v2:172.21.15.72:6805/2073231730' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-26T01:58:20.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:19 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:58:20.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:19 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:20.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:19 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:20.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:19 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:20.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:19 smithi173 ceph-mon[33839]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:20.164 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:58:20.218 INFO:journalctl@ceph.osd.1.smithi072.stdout:Mar 26 01:58:19 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-1[52314]: 2024-03-26T01:58:19.983+0000 7f39a627c700 -1 osd.1 0 waiting for initial osdmap 2024-03-26T01:58:20.219 INFO:journalctl@ceph.osd.1.smithi072.stdout:Mar 26 01:58:19 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-1[52314]: 2024-03-26T01:58:19.985+0000 7f399e067700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T01:58:20.768 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:20 smithi072 ceph-mon[30177]: from='osd.1 v2:172.21.15.72:6805/2073231730' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-03-26T01:58:20.769 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:20 smithi072 ceph-mon[30177]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T01:58:20.769 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:20 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:58:20.769 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:20 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:58:20.769 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:20 smithi072 ceph-mon[30177]: from='osd.1 v2:172.21.15.72:6805/2073231730' entity='osd.1' 2024-03-26T01:58:20.769 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:20 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:58:20.769 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:20 smithi072 ceph-mon[41619]: from='osd.1 v2:172.21.15.72:6805/2073231730' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-03-26T01:58:20.769 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:20 smithi072 ceph-mon[41619]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T01:58:20.769 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:20 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:58:20.769 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:20 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:58:20.769 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:20 smithi072 ceph-mon[41619]: from='osd.1 v2:172.21.15.72:6805/2073231730' entity='osd.1' 2024-03-26T01:58:20.769 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:20 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:58:21.062 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:20 smithi173 ceph-mon[33839]: from='osd.1 v2:172.21.15.72:6805/2073231730' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-03-26T01:58:21.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:20 smithi173 ceph-mon[33839]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T01:58:21.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:20 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:58:21.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:20 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:58:21.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:20 smithi173 ceph-mon[33839]: from='osd.1 v2:172.21.15.72:6805/2073231730' entity='osd.1' 2024-03-26T01:58:21.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:20 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:58:22.254 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:21 smithi072 ceph-mon[30177]: purged_snaps scrub starts 2024-03-26T01:58:22.254 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:21 smithi072 ceph-mon[30177]: purged_snaps scrub ok 2024-03-26T01:58:22.254 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:21 smithi072 ceph-mon[30177]: osd.1 v2:172.21.15.72:6805/2073231730 boot 2024-03-26T01:58:22.254 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:21 smithi072 ceph-mon[30177]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T01:58:22.254 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:21 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:58:22.254 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:21 smithi072 ceph-mon[30177]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:22.254 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:21 smithi072 ceph-mon[41619]: purged_snaps scrub starts 2024-03-26T01:58:22.254 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:21 smithi072 ceph-mon[41619]: purged_snaps scrub ok 2024-03-26T01:58:22.255 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:21 smithi072 ceph-mon[41619]: osd.1 v2:172.21.15.72:6805/2073231730 boot 2024-03-26T01:58:22.255 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:21 smithi072 ceph-mon[41619]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T01:58:22.255 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:21 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:58:22.255 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:21 smithi072 ceph-mon[41619]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:22.312 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:21 smithi173 ceph-mon[33839]: purged_snaps scrub starts 2024-03-26T01:58:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:21 smithi173 ceph-mon[33839]: purged_snaps scrub ok 2024-03-26T01:58:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:21 smithi173 ceph-mon[33839]: osd.1 v2:172.21.15.72:6805/2073231730 boot 2024-03-26T01:58:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:21 smithi173 ceph-mon[33839]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T01:58:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:21 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T01:58:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:21 smithi173 ceph-mon[33839]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T01:58:23.130 INFO:teuthology.orchestra.run.smithi072.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T01:58:23.130 INFO:teuthology.orchestra.run.smithi072.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T01:58:23.130 INFO:teuthology.orchestra.run.smithi072.stderr: stderr: 10+0 records in 2024-03-26T01:58:23.130 INFO:teuthology.orchestra.run.smithi072.stderr:10+0 records out 2024-03-26T01:58:23.130 INFO:teuthology.orchestra.run.smithi072.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112494 s, 932 MB/s 2024-03-26T01:58:23.130 INFO:teuthology.orchestra.run.smithi072.stderr:--> Zapping successful for: 2024-03-26T01:58:23.312 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:22 smithi173 ceph-mon[33839]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T01:58:23.468 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:22 smithi072 ceph-mon[41619]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T01:58:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:22 smithi072 ceph-mon[30177]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T01:58:23.760 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi072:vg_nvme/lv_2 2024-03-26T01:58:23.996 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:58:24.257 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:24 smithi072 ceph-mon[30177]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:24.257 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:24 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:24.257 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:24 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:24.258 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:24 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:58:24.258 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:24 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:58:24.258 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:24 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:24.258 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:24 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:24.258 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:24 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:24.258 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:24 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:24.258 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:24 smithi072 ceph-mon[41619]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:24.258 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:24 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:24.258 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:24 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:24.258 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:24 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:58:24.258 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:24 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:58:24.259 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:24 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:24.259 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:24 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:24.259 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:24 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:24.259 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:24 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:24.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:24 smithi173 ceph-mon[33839]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:24.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:24 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:24.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:24 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:24.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:24 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:58:24.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:24 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:58:24.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:24 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:24.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:24 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:24.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:24 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:24.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:24 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:25.140 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:25 smithi072 ceph-mon[30177]: Detected new or changed devices on smithi072 2024-03-26T01:58:25.140 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:25 smithi072 ceph-mon[30177]: Adjusting osd_memory_target on smithi072 to 8032M 2024-03-26T01:58:25.140 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:25 smithi072 ceph-mon[41619]: Detected new or changed devices on smithi072 2024-03-26T01:58:25.141 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:25 smithi072 ceph-mon[41619]: Adjusting osd_memory_target on smithi072 to 8032M 2024-03-26T01:58:25.312 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:25 smithi173 ceph-mon[33839]: Detected new or changed devices on smithi072 2024-03-26T01:58:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:25 smithi173 ceph-mon[33839]: Adjusting osd_memory_target on smithi072 to 8032M 2024-03-26T01:58:25.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.660+0000 7f163ff3a700 1 -- 172.21.15.72:0/2040165013 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1638103720 msgr2=0x7f1638103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:58:25.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.660+0000 7f163ff3a700 1 --2- 172.21.15.72:0/2040165013 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1638103720 0x7f1638103b40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f1634009990 tx=0x7f1634009ca0 comp rx=0 tx=0).stop 2024-03-26T01:58:25.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.660+0000 7f163ff3a700 1 -- 172.21.15.72:0/2040165013 shutdown_connections 2024-03-26T01:58:25.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.660+0000 7f163ff3a700 1 --2- 172.21.15.72:0/2040165013 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1638100b40 0x7f163810a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:58:25.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.660+0000 7f163ff3a700 1 --2- 172.21.15.72:0/2040165013 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1638103720 0x7f1638103b40 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:58:25.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.660+0000 7f163ff3a700 1 --2- 172.21.15.72:0/2040165013 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f16381001a0 0x7f1638100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:58:25.660 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.660+0000 7f163ff3a700 1 -- 172.21.15.72:0/2040165013 >> 172.21.15.72:0/2040165013 conn(0x7f16380fbd20 msgr2=0x7f16380fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:58:25.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.660+0000 7f163ff3a700 1 -- 172.21.15.72:0/2040165013 shutdown_connections 2024-03-26T01:58:25.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.660+0000 7f163ff3a700 1 -- 172.21.15.72:0/2040165013 wait complete. 2024-03-26T01:58:25.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.660+0000 7f163ff3a700 1 Processor -- start 2024-03-26T01:58:25.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.660+0000 7f163ff3a700 1 -- start start 2024-03-26T01:58:25.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.660+0000 7f163ff3a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16381001a0 0x7f1638102e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:58:25.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.660+0000 7f163ff3a700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1638100b40 0x7f1638101470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:58:25.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.660+0000 7f163ff3a700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1638103720 0x7f16381019b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:58:25.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.660+0000 7f163ff3a700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1638102130 con 0x7f1638103720 2024-03-26T01:58:25.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.660+0000 7f163ff3a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16381022a0 con 0x7f16381001a0 2024-03-26T01:58:25.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.660+0000 7f163ff3a700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16381a1710 con 0x7f1638100b40 2024-03-26T01:58:25.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.661+0000 7f163d4d5700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1638100b40 0x7f1638101470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:58:25.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.661+0000 7f163e4d7700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1638103720 0x7f16381019b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:58:25.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.661+0000 7f163d4d5700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1638100b40 0x7f1638101470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:51002/0 (socket says 172.21.15.72:51002) 2024-03-26T01:58:25.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.661+0000 7f163d4d5700 1 -- 172.21.15.72:0/3007389025 learned_addr learned my addr 172.21.15.72:0/3007389025 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:58:25.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.661+0000 7f163dcd6700 1 --2- 172.21.15.72:0/3007389025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16381001a0 0x7f1638102e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:58:25.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.661+0000 7f163d4d5700 1 -- 172.21.15.72:0/3007389025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16381001a0 msgr2=0x7f1638102e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:58:25.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.661+0000 7f163d4d5700 1 --2- 172.21.15.72:0/3007389025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16381001a0 0x7f1638102e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:58:25.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.661+0000 7f163d4d5700 1 -- 172.21.15.72:0/3007389025 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1638103720 msgr2=0x7f16381019b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:58:25.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.661+0000 7f163d4d5700 1 --2- 172.21.15.72:0/3007389025 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1638103720 0x7f16381019b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:58:25.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.661+0000 7f163d4d5700 1 -- 172.21.15.72:0/3007389025 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16340095f0 con 0x7f1638100b40 2024-03-26T01:58:25.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.661+0000 7f163e4d7700 1 --2- 172.21.15.72:0/3007389025 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1638103720 0x7f16381019b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T01:58:25.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.661+0000 7f163d4d5700 1 --2- 172.21.15.72:0/3007389025 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1638100b40 0x7f1638101470 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f162800e9a0 tx=0x7f162800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:58:25.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.661+0000 7f162effd700 1 -- 172.21.15.72:0/3007389025 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1628009790 con 0x7f1638100b40 2024-03-26T01:58:25.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.661+0000 7f163ff3a700 1 -- 172.21.15.72:0/3007389025 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16381a19f0 con 0x7f1638100b40 2024-03-26T01:58:25.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.661+0000 7f162effd700 1 -- 172.21.15.72:0/3007389025 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f162800cbd0 con 0x7f1638100b40 2024-03-26T01:58:25.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.661+0000 7f162effd700 1 -- 172.21.15.72:0/3007389025 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1628018680 con 0x7f1638100b40 2024-03-26T01:58:25.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.661+0000 7f163ff3a700 1 -- 172.21.15.72:0/3007389025 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16381a1f40 con 0x7f1638100b40 2024-03-26T01:58:25.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.662+0000 7f163ff3a700 1 -- 172.21.15.72:0/3007389025 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1638109620 con 0x7f1638100b40 2024-03-26T01:58:25.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.663+0000 7f162effd700 1 -- 172.21.15.72:0/3007389025 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 14) v1 ==== 91307+0+0 (secure 0 0 0) 0x7f1628010430 con 0x7f1638100b40 2024-03-26T01:58:25.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.663+0000 7f162effd700 1 --2- 172.21.15.72:0/3007389025 >> v2:172.21.15.72:6800/2923394687 conn(0x7f162406d900 0x7f162406fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:58:25.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.663+0000 7f162effd700 1 -- 172.21.15.72:0/3007389025 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 1833+0+0 (secure 0 0 0) 0x7f1628014070 con 0x7f1638100b40 2024-03-26T01:58:25.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.663+0000 7f163dcd6700 1 --2- 172.21.15.72:0/3007389025 >> v2:172.21.15.72:6800/2923394687 conn(0x7f162406d900 0x7f162406fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:58:25.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.664+0000 7f163dcd6700 1 --2- 172.21.15.72:0/3007389025 >> v2:172.21.15.72:6800/2923394687 conn(0x7f162406d900 0x7f162406fdc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f163400b760 tx=0x7f163400b6f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:58:25.668 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.668+0000 7f162effd700 1 -- 172.21.15.72:0/3007389025 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1628053280 con 0x7f1638100b40 2024-03-26T01:58:25.871 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:25.870+0000 7f163ff3a700 1 -- 172.21.15.72:0/3007389025 --> v2:172.21.15.72:6800/2923394687 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f16380624c0 con 0x7f162406d900 2024-03-26T01:58:26.312 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:26 smithi173 ceph-mon[33839]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:26.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:26 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:58:26.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:26 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:58:26.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:26 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:26.464 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:26 smithi072 ceph-mon[30177]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:26.464 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:26 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:58:26.465 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:26 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:58:26.465 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:26 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:26.465 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:26 smithi072 ceph-mon[41619]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:26.465 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:26 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:58:26.465 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:26 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:58:26.465 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:26 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:27.312 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:27 smithi173 ceph-mon[33839]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:27.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:27 smithi072 ceph-mon[30177]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:27.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:27 smithi072 ceph-mon[41619]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:28.312 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:28 smithi173 ceph-mon[33839]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:28.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:28 smithi072 ceph-mon[30177]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:28.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:28 smithi072 ceph-mon[41619]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:29.218 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:29 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3300592596' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65c5c06e-e84d-4b5a-8204-65399b224f1b"}]: dispatch 2024-03-26T01:58:29.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:29 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3300592596' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "65c5c06e-e84d-4b5a-8204-65399b224f1b"}]': finished 2024-03-26T01:58:29.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:29 smithi072 ceph-mon[30177]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T01:58:29.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:29 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:29.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:29 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3300592596' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65c5c06e-e84d-4b5a-8204-65399b224f1b"}]: dispatch 2024-03-26T01:58:29.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:29 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3300592596' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "65c5c06e-e84d-4b5a-8204-65399b224f1b"}]': finished 2024-03-26T01:58:29.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:29 smithi072 ceph-mon[41619]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T01:58:29.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:29 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:29.312 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:29 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3300592596' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "65c5c06e-e84d-4b5a-8204-65399b224f1b"}]: dispatch 2024-03-26T01:58:29.312 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:29 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3300592596' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "65c5c06e-e84d-4b5a-8204-65399b224f1b"}]': finished 2024-03-26T01:58:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:29 smithi173 ceph-mon[33839]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T01:58:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:29 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:30.312 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:30 smithi173 ceph-mon[33839]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:30.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1489193887' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:58:30.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:30 smithi072 ceph-mon[30177]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:30.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1489193887' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:58:30.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:30 smithi072 ceph-mon[41619]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:30.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1489193887' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:58:32.247 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:32 smithi072 ceph-mon[30177]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:32.248 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:32 smithi072 ceph-mon[41619]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:32.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:32 smithi173 ceph-mon[33839]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:34.186 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:34 smithi072 ceph-mon[30177]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:34.186 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:34 smithi072 ceph-mon[41619]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:34.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:34 smithi173 ceph-mon[33839]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:36.468 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:36 smithi072 ceph-mon[41619]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:36.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:36 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T01:58:36.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:36 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:36.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:36 smithi072 ceph-mon[30177]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:36.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:36 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T01:58:36.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:36 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:36.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:36 smithi173 ceph-mon[33839]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:36.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:36 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T01:58:36.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:36 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:37.421 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:37 smithi072 ceph-mon[30177]: Deploying daemon osd.2 on smithi072 2024-03-26T01:58:37.422 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:37 smithi072 ceph-mon[41619]: Deploying daemon osd.2 on smithi072 2024-03-26T01:58:37.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:37 smithi173 ceph-mon[33839]: Deploying daemon osd.2 on smithi072 2024-03-26T01:58:38.195 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:38 smithi072 ceph-mon[41619]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:38.195 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:38 smithi072 ceph-mon[30177]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:38.464 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:38 smithi173 ceph-mon[33839]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:40.218 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:40 smithi072 ceph-mon[30177]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:40.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:40 smithi072 ceph-mon[41619]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:40.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:40 smithi173 ceph-mon[33839]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:42.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:41 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:42.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:41 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:42.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:41 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:42.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:41 smithi173 ceph-mon[33839]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:42.110 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:41 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:42.110 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:41 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:42.110 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:41 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:42.111 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:41 smithi072 ceph-mon[30177]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:42.111 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:41 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:42.111 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:41 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:42.111 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:41 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:42.111 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:41 smithi072 ceph-mon[41619]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:44.224 INFO:teuthology.orchestra.run.smithi072.stdout:Created osd(s) 2 on host 'smithi072' 2024-03-26T01:58:44.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:44.223+0000 7f162effd700 1 -- 172.21.15.72:0/3007389025 <== mgr.14152 v2:172.21.15.72:6800/2923394687 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f16380624c0 con 0x7f162406d900 2024-03-26T01:58:44.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:44.226+0000 7f163ff3a700 1 -- 172.21.15.72:0/3007389025 >> v2:172.21.15.72:6800/2923394687 conn(0x7f162406d900 msgr2=0x7f162406fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:58:44.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:44.226+0000 7f163ff3a700 1 --2- 172.21.15.72:0/3007389025 >> v2:172.21.15.72:6800/2923394687 conn(0x7f162406d900 0x7f162406fdc0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f163400b760 tx=0x7f163400b6f0 comp rx=0 tx=0).stop 2024-03-26T01:58:44.226 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:44.226+0000 7f163ff3a700 1 -- 172.21.15.72:0/3007389025 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1638100b40 msgr2=0x7f1638101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:58:44.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:44.226+0000 7f163ff3a700 1 --2- 172.21.15.72:0/3007389025 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1638100b40 0x7f1638101470 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f162800e9a0 tx=0x7f162800ecb0 comp rx=0 tx=0).stop 2024-03-26T01:58:44.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:44.226+0000 7f163ff3a700 1 -- 172.21.15.72:0/3007389025 shutdown_connections 2024-03-26T01:58:44.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:44.226+0000 7f163ff3a700 1 --2- 172.21.15.72:0/3007389025 >> v2:172.21.15.72:6800/2923394687 conn(0x7f162406d900 0x7f162406fdc0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:58:44.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:44.226+0000 7f163ff3a700 1 --2- 172.21.15.72:0/3007389025 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1638103720 0x7f16381019b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:58:44.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:44.226+0000 7f163ff3a700 1 --2- 172.21.15.72:0/3007389025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f16381001a0 0x7f1638102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:58:44.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:44.226+0000 7f163ff3a700 1 --2- 172.21.15.72:0/3007389025 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1638100b40 0x7f1638101470 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:58:44.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:44.226+0000 7f163ff3a700 1 -- 172.21.15.72:0/3007389025 >> 172.21.15.72:0/3007389025 conn(0x7f16380fbd20 msgr2=0x7f1638106760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:58:44.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:44.226+0000 7f163ff3a700 1 -- 172.21.15.72:0/3007389025 shutdown_connections 2024-03-26T01:58:44.227 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:44.226+0000 7f163ff3a700 1 -- 172.21.15.72:0/3007389025 wait complete. 2024-03-26T01:58:44.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:44 smithi072 ceph-mon[30177]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:44.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:44 smithi072 ceph-mon[30177]: from='osd.2 v2:172.21.15.72:6809/3459293761' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T01:58:44.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:44 smithi072 ceph-mon[30177]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T01:58:44.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:44 smithi072 ceph-mon[41619]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:44.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:44 smithi072 ceph-mon[41619]: from='osd.2 v2:172.21.15.72:6809/3459293761' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T01:58:44.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:44 smithi072 ceph-mon[41619]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T01:58:44.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:44 smithi173 ceph-mon[33839]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:44.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:44 smithi173 ceph-mon[33839]: from='osd.2 v2:172.21.15.72:6809/3459293761' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T01:58:44.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:44 smithi173 ceph-mon[33839]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T01:58:44.846 DEBUG:teuthology.orchestra.run.smithi072:osd.2> sudo journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.2.service 2024-03-26T01:58:44.848 INFO:tasks.cephadm:Deploying osd.3 on smithi072 with /dev/vg_nvme/lv_1... 2024-03-26T01:58:44.848 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T01:58:45.145 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:58:45.415 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:45 smithi072 ceph-mon[30177]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T01:58:45.416 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:45 smithi072 ceph-mon[30177]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T01:58:45.416 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:45 smithi072 ceph-mon[30177]: from='osd.2 v2:172.21.15.72:6809/3459293761' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-26T01:58:45.416 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:45 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:45.416 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:45 smithi072 ceph-mon[30177]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-26T01:58:45.416 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:45 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:45.416 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:45 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:45.416 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:45 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:45.416 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:45 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:45.416 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:45 smithi072 ceph-mon[30177]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-03-26T01:58:45.416 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:45 smithi072 ceph-mon[41619]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T01:58:45.417 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:45 smithi072 ceph-mon[41619]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T01:58:45.417 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:45 smithi072 ceph-mon[41619]: from='osd.2 v2:172.21.15.72:6809/3459293761' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-26T01:58:45.417 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:45 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:45.417 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:45 smithi072 ceph-mon[41619]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-26T01:58:45.417 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:45 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:45.417 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:45 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:45.417 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:45 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:45.417 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:45 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:45.417 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:45 smithi072 ceph-mon[41619]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-03-26T01:58:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:45 smithi173 ceph-mon[33839]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T01:58:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:45 smithi173 ceph-mon[33839]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T01:58:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:45 smithi173 ceph-mon[33839]: from='osd.2 v2:172.21.15.72:6809/3459293761' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-26T01:58:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:45 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:45 smithi173 ceph-mon[33839]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-26T01:58:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:45 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:45 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:45 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:45 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:45 smithi173 ceph-mon[33839]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-03-26T01:58:45.968 INFO:journalctl@ceph.osd.2.smithi072.stdout:Mar 26 01:58:45 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-2[58463]: 2024-03-26T01:58:45.613+0000 7f43ea469700 -1 osd.2 0 waiting for initial osdmap 2024-03-26T01:58:45.969 INFO:journalctl@ceph.osd.2.smithi072.stdout:Mar 26 01:58:45 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-2[58463]: 2024-03-26T01:58:45.616+0000 7f43e2254700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T01:58:46.277 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:46 smithi072 ceph-mon[30177]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:46.277 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:46 smithi072 ceph-mon[30177]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T01:58:46.277 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:46 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:46.277 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:46 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:46.278 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:46 smithi072 ceph-mon[30177]: from='osd.2 ' entity='osd.2' 2024-03-26T01:58:46.278 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:46 smithi072 ceph-mon[41619]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:46.278 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:46 smithi072 ceph-mon[41619]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T01:58:46.278 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:46 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:46.278 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:46 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:46.278 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:46 smithi072 ceph-mon[41619]: from='osd.2 ' entity='osd.2' 2024-03-26T01:58:46.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:46 smithi173 ceph-mon[33839]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:46.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:46 smithi173 ceph-mon[33839]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T01:58:46.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:46 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:46.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:46 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:46.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:46 smithi173 ceph-mon[33839]: from='osd.2 ' entity='osd.2' 2024-03-26T01:58:47.405 INFO:teuthology.orchestra.run.smithi072.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T01:58:47.405 INFO:teuthology.orchestra.run.smithi072.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T01:58:47.406 INFO:teuthology.orchestra.run.smithi072.stderr: stderr: 10+0 records in 2024-03-26T01:58:47.406 INFO:teuthology.orchestra.run.smithi072.stderr:10+0 records out 2024-03-26T01:58:47.406 INFO:teuthology.orchestra.run.smithi072.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0125947 s, 833 MB/s 2024-03-26T01:58:47.406 INFO:teuthology.orchestra.run.smithi072.stderr:--> Zapping successful for: 2024-03-26T01:58:47.718 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:47 smithi072 ceph-mon[30177]: purged_snaps scrub starts 2024-03-26T01:58:47.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:47 smithi072 ceph-mon[30177]: purged_snaps scrub ok 2024-03-26T01:58:47.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:47 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:47.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:47 smithi072 ceph-mon[30177]: osd.2 v2:172.21.15.72:6809/3459293761 boot 2024-03-26T01:58:47.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:47 smithi072 ceph-mon[30177]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T01:58:47.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:47 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:47.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:47 smithi072 ceph-mon[41619]: purged_snaps scrub starts 2024-03-26T01:58:47.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:47 smithi072 ceph-mon[41619]: purged_snaps scrub ok 2024-03-26T01:58:47.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:47 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:47.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:47 smithi072 ceph-mon[41619]: osd.2 v2:172.21.15.72:6809/3459293761 boot 2024-03-26T01:58:47.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:47 smithi072 ceph-mon[41619]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T01:58:47.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:47 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:47.812 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:47 smithi173 ceph-mon[33839]: purged_snaps scrub starts 2024-03-26T01:58:47.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:47 smithi173 ceph-mon[33839]: purged_snaps scrub ok 2024-03-26T01:58:47.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:47 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:47.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:47 smithi173 ceph-mon[33839]: osd.2 v2:172.21.15.72:6809/3459293761 boot 2024-03-26T01:58:47.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:47 smithi173 ceph-mon[33839]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T01:58:47.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:47 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T01:58:47.964 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi072:vg_nvme/lv_1 2024-03-26T01:58:48.439 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T01:58:48.670 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:48 smithi072 ceph-mon[30177]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:48.670 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:48 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T01:58:48.670 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:48 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T01:58:48.670 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:48 smithi072 ceph-mon[30177]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T01:58:48.670 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:48 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T01:58:48.671 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:48 smithi072 ceph-mon[41619]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:48.671 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:48 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T01:58:48.671 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:48 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T01:58:48.671 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:48 smithi072 ceph-mon[41619]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T01:58:48.671 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:48 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T01:58:48.812 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:48 smithi173 ceph-mon[33839]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T01:58:48.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:48 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T01:58:48.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:48 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T01:58:48.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:48 smithi173 ceph-mon[33839]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T01:58:48.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:48 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T01:58:49.221 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 26 01:58:48 smithi072 sudo[60249]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T01:58:49.221 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 26 01:58:48 smithi072 sudo[60249]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T01:58:49.221 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 26 01:58:48 smithi072 sudo[60249]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T01:58:49.221 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 26 01:58:48 smithi072 sudo[60249]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:58:49.505 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 26 01:58:49 smithi072 sudo[60280]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T01:58:49.506 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 26 01:58:49 smithi072 sudo[60280]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T01:58:49.506 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 26 01:58:49 smithi072 sudo[60280]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T01:58:49.506 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 26 01:58:49 smithi072 sudo[60280]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:58:49.771 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:49 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T01:58:49.771 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:49 smithi072 ceph-mon[30177]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T01:58:49.771 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:49 smithi072 ceph-mon[30177]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:49.771 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:49 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:49.772 INFO:journalctl@ceph.osd.1.smithi072.stdout:Mar 26 01:58:49 smithi072 sudo[60554]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T01:58:49.772 INFO:journalctl@ceph.osd.1.smithi072.stdout:Mar 26 01:58:49 smithi072 sudo[60554]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T01:58:49.772 INFO:journalctl@ceph.osd.1.smithi072.stdout:Mar 26 01:58:49 smithi072 sudo[60554]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T01:58:49.772 INFO:journalctl@ceph.osd.1.smithi072.stdout:Mar 26 01:58:49 smithi072 sudo[60554]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:58:49.772 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:49 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T01:58:49.772 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:49 smithi072 ceph-mon[41619]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T01:58:49.772 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:49 smithi072 ceph-mon[41619]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:49.772 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:49 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:50.031 INFO:journalctl@ceph.osd.1.smithi072.stdout:Mar 26 01:58:49 smithi072 sudo[62841]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T01:58:50.031 INFO:journalctl@ceph.osd.1.smithi072.stdout:Mar 26 01:58:49 smithi072 sudo[62841]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T01:58:50.031 INFO:journalctl@ceph.osd.1.smithi072.stdout:Mar 26 01:58:49 smithi072 sudo[62841]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T01:58:50.031 INFO:journalctl@ceph.osd.1.smithi072.stdout:Mar 26 01:58:49 smithi072 sudo[62841]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:58:50.062 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:49 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T01:58:50.062 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:49 smithi173 ceph-mon[33839]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T01:58:50.062 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:49 smithi173 ceph-mon[33839]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:50.062 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:49 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:50.287 INFO:journalctl@ceph.osd.2.smithi072.stdout:Mar 26 01:58:50 smithi072 sudo[62862]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T01:58:50.287 INFO:journalctl@ceph.osd.2.smithi072.stdout:Mar 26 01:58:50 smithi072 sudo[62862]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T01:58:50.287 INFO:journalctl@ceph.osd.2.smithi072.stdout:Mar 26 01:58:50 smithi072 sudo[62862]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T01:58:50.287 INFO:journalctl@ceph.osd.2.smithi072.stdout:Mar 26 01:58:50 smithi072 sudo[62862]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:58:50.550 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:50 smithi072 sudo[62877]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T01:58:50.550 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:50 smithi072 sudo[62877]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T01:58:50.550 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:50 smithi072 sudo[62877]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T01:58:50.550 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:50 smithi072 sudo[62877]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:58:50.551 INFO:journalctl@ceph.osd.2.smithi072.stdout:Mar 26 01:58:50 smithi072 sudo[62868]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T01:58:50.551 INFO:journalctl@ceph.osd.2.smithi072.stdout:Mar 26 01:58:50 smithi072 sudo[62868]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T01:58:50.551 INFO:journalctl@ceph.osd.2.smithi072.stdout:Mar 26 01:58:50 smithi072 sudo[62868]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T01:58:50.551 INFO:journalctl@ceph.osd.2.smithi072.stdout:Mar 26 01:58:50 smithi072 sudo[62868]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:58:51.062 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:50 smithi173 ceph-mon[33839]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T01:58:51.062 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:50 smithi173 ceph-mon[33839]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T01:58:51.062 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 sudo[36620]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T01:58:51.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 sudo[36620]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T01:58:51.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 sudo[36620]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T01:58:51.076 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:50 smithi072 ceph-mon[41619]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T01:58:51.076 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:50 smithi072 ceph-mon[41619]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T01:58:51.077 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:50 smithi072 sudo[62882]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T01:58:51.077 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:50 smithi072 sudo[62882]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T01:58:51.077 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:50 smithi072 sudo[62882]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T01:58:51.077 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:50 smithi072 sudo[62882]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:58:51.077 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:50 smithi072 ceph-mon[30177]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T01:58:51.077 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:50 smithi072 ceph-mon[30177]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T01:58:51.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.107+0000 7ffb0a533700 1 -- 172.21.15.72:0/1368362318 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffb041020b0 msgr2=0x7ffb041024d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:58:51.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.107+0000 7ffb0a533700 1 --2- 172.21.15.72:0/1368362318 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffb041020b0 0x7ffb041024d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ffaf40098f0 tx=0x7ffaf4009c00 comp rx=0 tx=0).stop 2024-03-26T01:58:51.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.108+0000 7ffb0a533700 1 -- 172.21.15.72:0/1368362318 shutdown_connections 2024-03-26T01:58:51.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.108+0000 7ffb0a533700 1 --2- 172.21.15.72:0/1368362318 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb04103c50 0x7ffb0410a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:58:51.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.108+0000 7ffb0a533700 1 --2- 172.21.15.72:0/1368362318 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffb041020b0 0x7ffb041024d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:58:51.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.108+0000 7ffb0a533700 1 --2- 172.21.15.72:0/1368362318 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb041032b0 0x7ffb04103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:58:51.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.108+0000 7ffb0a533700 1 -- 172.21.15.72:0/1368362318 >> 172.21.15.72:0/1368362318 conn(0x7ffb040fd690 msgr2=0x7ffb040ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:58:51.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.108+0000 7ffb0a533700 1 -- 172.21.15.72:0/1368362318 shutdown_connections 2024-03-26T01:58:51.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.108+0000 7ffb0a533700 1 -- 172.21.15.72:0/1368362318 wait complete. 2024-03-26T01:58:51.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.108+0000 7ffb0a533700 1 Processor -- start 2024-03-26T01:58:51.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.108+0000 7ffb0a533700 1 -- start start 2024-03-26T01:58:51.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.108+0000 7ffb0a533700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb041020b0 0x7ffb04197870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:58:51.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.108+0000 7ffb0a533700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb041032b0 0x7ffb04197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:58:51.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.108+0000 7ffb0a533700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffb04103c50 0x7ffb0419ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:58:51.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.108+0000 7ffb0a533700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0419f3b0 con 0x7ffb041032b0 2024-03-26T01:58:51.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.108+0000 7ffb0a533700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0419f4f0 con 0x7ffb041020b0 2024-03-26T01:58:51.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.108+0000 7ffb0a533700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb0419f680 con 0x7ffb04103c50 2024-03-26T01:58:51.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.109+0000 7ffb037fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb041032b0 0x7ffb04197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:58:51.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.109+0000 7ffb037fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb041032b0 0x7ffb04197db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:45364/0 (socket says 172.21.15.72:45364) 2024-03-26T01:58:51.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.109+0000 7ffb037fe700 1 -- 172.21.15.72:0/1877411221 learned_addr learned my addr 172.21.15.72:0/1877411221 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T01:58:51.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.109+0000 7ffb08ad0700 1 --2- 172.21.15.72:0/1877411221 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffb04103c50 0x7ffb0419ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:58:51.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.109+0000 7ffb03fff700 1 --2- 172.21.15.72:0/1877411221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb041020b0 0x7ffb04197870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:58:51.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.109+0000 7ffb037fe700 1 -- 172.21.15.72:0/1877411221 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffb04103c50 msgr2=0x7ffb0419ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:58:51.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.109+0000 7ffb037fe700 1 --2- 172.21.15.72:0/1877411221 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffb04103c50 0x7ffb0419ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:58:51.111 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.109+0000 7ffb037fe700 1 -- 172.21.15.72:0/1877411221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb041020b0 msgr2=0x7ffb04197870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:58:51.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.109+0000 7ffb037fe700 1 --2- 172.21.15.72:0/1877411221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb041020b0 0x7ffb04197870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:58:51.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.109+0000 7ffb037fe700 1 -- 172.21.15.72:0/1877411221 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf80095f0 con 0x7ffb041032b0 2024-03-26T01:58:51.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.109+0000 7ffb037fe700 1 --2- 172.21.15.72:0/1877411221 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb041032b0 0x7ffb04197db0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ffaf800ec40 tx=0x7ffaf800ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:58:51.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.109+0000 7ffb017fa700 1 -- 172.21.15.72:0/1877411221 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf8009a10 con 0x7ffb041032b0 2024-03-26T01:58:51.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.109+0000 7ffb0a533700 1 -- 172.21.15.72:0/1877411221 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf4009520 con 0x7ffb041032b0 2024-03-26T01:58:51.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.109+0000 7ffb017fa700 1 -- 172.21.15.72:0/1877411221 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7ffaf80044d0 con 0x7ffb041032b0 2024-03-26T01:58:51.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.109+0000 7ffb017fa700 1 -- 172.21.15.72:0/1877411221 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf8003ea0 con 0x7ffb041032b0 2024-03-26T01:58:51.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.109+0000 7ffb0a533700 1 -- 172.21.15.72:0/1877411221 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb0419fcd0 con 0x7ffb041032b0 2024-03-26T01:58:51.112 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.110+0000 7ffaeaffd700 1 -- 172.21.15.72:0/1877411221 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffae4005120 con 0x7ffb041032b0 2024-03-26T01:58:51.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.111+0000 7ffb017fa700 1 -- 172.21.15.72:0/1877411221 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 15) v1 ==== 91363+0+0 (secure 0 0 0) 0x7ffaf8004000 con 0x7ffb041032b0 2024-03-26T01:58:51.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.111+0000 7ffb017fa700 1 --2- 172.21.15.72:0/1877411221 >> v2:172.21.15.72:6800/2923394687 conn(0x7ffaec06d990 0x7ffaec06fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:58:51.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.111+0000 7ffb03fff700 1 --2- 172.21.15.72:0/1877411221 >> v2:172.21.15.72:6800/2923394687 conn(0x7ffaec06d990 0x7ffaec06fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:58:51.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.111+0000 7ffb017fa700 1 -- 172.21.15.72:0/1877411221 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 2511+0+0 (secure 0 0 0) 0x7ffaf8014070 con 0x7ffb041032b0 2024-03-26T01:58:51.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.112+0000 7ffb03fff700 1 --2- 172.21.15.72:0/1877411221 >> v2:172.21.15.72:6800/2923394687 conn(0x7ffaec06d990 0x7ffaec06fe50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ffaf40098c0 tx=0x7ffaf4019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:58:51.116 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.116+0000 7ffb017fa700 1 -- 172.21.15.72:0/1877411221 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ffaf8057100 con 0x7ffb041032b0 2024-03-26T01:58:51.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:58:51.325+0000 7ffaeaffd700 1 -- 172.21.15.72:0/1877411221 --> v2:172.21.15.72:6800/2923394687 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7ffae4000bc0 con 0x7ffaec06d990 2024-03-26T01:58:51.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 sudo[36620]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:58:51.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 sudo[36623]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T01:58:51.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 sudo[36623]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T01:58:51.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 sudo[36623]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T01:58:51.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 sudo[36623]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:58:51.857 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 sudo[62998]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T01:58:51.857 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 sudo[62998]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T01:58:51.857 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 sudo[62998]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T01:58:51.857 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 sudo[62998]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:58:52.185 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: Detected new or changed devices on smithi072 2024-03-26T01:58:52.185 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T01:58:52.185 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:58:52.185 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:58:52.185 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:58:52.185 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T01:58:52.185 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T01:58:52.185 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:58:52.185 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:58:52.185 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:58:52.185 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:52.185 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T01:58:52.186 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:52.186 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:52.186 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:58:52.186 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:58:52.186 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:58:52.186 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:58:52.186 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:52.186 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:58:52.186 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:52.186 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:52.186 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:52.186 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:58:52.186 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:58:52.187 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:58:52.187 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:52.187 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 sudo[63055]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T01:58:52.187 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 sudo[63055]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T01:58:52.187 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 sudo[63055]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T01:58:52.187 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 sudo[63055]: pam_unix(sudo:session): session closed for user root 2024-03-26T01:58:52.187 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: Detected new or changed devices on smithi072 2024-03-26T01:58:52.187 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T01:58:52.187 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:58:52.187 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:58:52.188 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:58:52.188 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T01:58:52.188 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T01:58:52.188 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:58:52.188 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:58:52.188 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:58:52.188 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:52.188 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T01:58:52.188 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:52.188 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:52.188 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:58:52.188 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:58:52.188 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:58:52.188 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:58:52.188 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:52.189 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:58:52.189 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:52.189 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:52.189 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:52.189 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:58:52.189 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:58:52.189 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:58:52.189 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:51 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: Detected new or changed devices on smithi072 2024-03-26T01:58:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T01:58:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:58:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:58:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:58:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T01:58:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T01:58:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:58:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:58:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:58:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: pgmap v83: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T01:58:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:58:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:58:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:58:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:58:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:58:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T01:58:52.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T01:58:52.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T01:58:52.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:51 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:52.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:52 smithi072 ceph-mon[30177]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:52.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:52 smithi072 ceph-mon[30177]: Adjusting osd_memory_target on smithi072 to 5354M 2024-03-26T01:58:52.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:52 smithi072 ceph-mon[30177]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T01:58:52.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:52 smithi072 ceph-mon[30177]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T01:58:52.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:52 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:52.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:52 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:52.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:52 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:52.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:52 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:52.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:52 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:52.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:52 smithi072 ceph-mon[41619]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:52.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:52 smithi072 ceph-mon[41619]: Adjusting osd_memory_target on smithi072 to 5354M 2024-03-26T01:58:52.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:52 smithi072 ceph-mon[41619]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T01:58:52.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:52 smithi072 ceph-mon[41619]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T01:58:52.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:52 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:52.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:52 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:52.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:52 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:52.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:52 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:52.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:52 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:53.312 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:52 smithi173 ceph-mon[33839]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi072:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:58:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:52 smithi173 ceph-mon[33839]: Adjusting osd_memory_target on smithi072 to 5354M 2024-03-26T01:58:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:52 smithi173 ceph-mon[33839]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T01:58:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:52 smithi173 ceph-mon[33839]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T01:58:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:52 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:52 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:58:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:52 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:58:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:52 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:58:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:52 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:58:54.218 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:53 smithi072 ceph-mon[30177]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:54.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:53 smithi072 ceph-mon[41619]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:54.312 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:53 smithi173 ceph-mon[33839]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:55.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:54 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4141417507' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f2f428c-7a84-4a34-8f42-179cbf06778d"}]: dispatch 2024-03-26T01:58:55.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:54 smithi072 ceph-mon[30177]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f2f428c-7a84-4a34-8f42-179cbf06778d"}]: dispatch 2024-03-26T01:58:55.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:54 smithi072 ceph-mon[30177]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f2f428c-7a84-4a34-8f42-179cbf06778d"}]': finished 2024-03-26T01:58:55.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:54 smithi072 ceph-mon[30177]: osdmap e22: 4 total, 3 up, 4 in 2024-03-26T01:58:55.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:54 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:58:55.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:54 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1843942988' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:58:55.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:54 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4141417507' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f2f428c-7a84-4a34-8f42-179cbf06778d"}]: dispatch 2024-03-26T01:58:55.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:54 smithi072 ceph-mon[41619]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f2f428c-7a84-4a34-8f42-179cbf06778d"}]: dispatch 2024-03-26T01:58:55.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:54 smithi072 ceph-mon[41619]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f2f428c-7a84-4a34-8f42-179cbf06778d"}]': finished 2024-03-26T01:58:55.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:54 smithi072 ceph-mon[41619]: osdmap e22: 4 total, 3 up, 4 in 2024-03-26T01:58:55.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:54 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:58:55.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:54 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1843942988' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:58:55.312 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:54 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4141417507' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f2f428c-7a84-4a34-8f42-179cbf06778d"}]: dispatch 2024-03-26T01:58:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:54 smithi173 ceph-mon[33839]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f2f428c-7a84-4a34-8f42-179cbf06778d"}]: dispatch 2024-03-26T01:58:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:54 smithi173 ceph-mon[33839]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f2f428c-7a84-4a34-8f42-179cbf06778d"}]': finished 2024-03-26T01:58:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:54 smithi173 ceph-mon[33839]: osdmap e22: 4 total, 3 up, 4 in 2024-03-26T01:58:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:54 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:58:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:54 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1843942988' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:58:56.218 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:55 smithi072 ceph-mon[30177]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:56.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:55 smithi072 ceph-mon[41619]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:56.312 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:55 smithi173 ceph-mon[33839]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:58:58.429 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:58:58 smithi072 ceph-mon[30177]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2024-03-26T01:58:58.429 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:58:58 smithi072 ceph-mon[41619]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2024-03-26T01:58:58.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:58:58 smithi173 ceph-mon[33839]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 65 KiB/s wr, 5 op/s 2024-03-26T01:59:00.425 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:00 smithi072 ceph-mon[30177]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-03-26T01:59:00.425 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:00 smithi072 ceph-mon[41619]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-03-26T01:59:00.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:00 smithi173 ceph-mon[33839]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2024-03-26T01:59:01.413 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:01 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T01:59:01.414 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:01 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:01.414 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:01 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T01:59:01.414 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:01 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:01.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:01 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T01:59:01.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:01 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:02.377 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:02 smithi072 ceph-mon[30177]: Deploying daemon osd.3 on smithi072 2024-03-26T01:59:02.377 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:02 smithi072 ceph-mon[30177]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-03-26T01:59:02.377 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:02 smithi072 ceph-mon[41619]: Deploying daemon osd.3 on smithi072 2024-03-26T01:59:02.377 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:02 smithi072 ceph-mon[41619]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-03-26T01:59:02.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:02 smithi173 ceph-mon[33839]: Deploying daemon osd.3 on smithi072 2024-03-26T01:59:02.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:02 smithi173 ceph-mon[33839]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2024-03-26T01:59:04.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:04 smithi072 ceph-mon[30177]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-26T01:59:04.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:04 smithi072 ceph-mon[41619]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-26T01:59:04.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:04 smithi173 ceph-mon[33839]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2024-03-26T01:59:06.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:06 smithi072 ceph-mon[30177]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 840 B/s rd, 24 KiB/s wr, 2 op/s 2024-03-26T01:59:06.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:06 smithi072 ceph-mon[41619]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 840 B/s rd, 24 KiB/s wr, 2 op/s 2024-03-26T01:59:06.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:06 smithi173 ceph-mon[33839]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 840 B/s rd, 24 KiB/s wr, 2 op/s 2024-03-26T01:59:07.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:07 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:07 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:07 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:59:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:07 smithi173 ceph-mon[33839]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-26T01:59:07.679 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:07 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:07.679 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:07 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:07.679 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:07 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:59:07.680 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:07 smithi072 ceph-mon[41619]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-26T01:59:07.680 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:07 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:07.680 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:07 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:07.680 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:07 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:59:07.680 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:07 smithi072 ceph-mon[30177]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-26T01:59:09.934 INFO:teuthology.orchestra.run.smithi072.stdout:Created osd(s) 3 on host 'smithi072' 2024-03-26T01:59:09.935 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:59:09.934+0000 7ffb017fa700 1 -- 172.21.15.72:0/1877411221 <== mgr.14152 v2:172.21.15.72:6800/2923394687 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ffae4000bc0 con 0x7ffaec06d990 2024-03-26T01:59:09.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:59:09.936+0000 7ffaeaffd700 1 -- 172.21.15.72:0/1877411221 >> v2:172.21.15.72:6800/2923394687 conn(0x7ffaec06d990 msgr2=0x7ffaec06fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:59:09.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:59:09.936+0000 7ffaeaffd700 1 --2- 172.21.15.72:0/1877411221 >> v2:172.21.15.72:6800/2923394687 conn(0x7ffaec06d990 0x7ffaec06fe50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ffaf40098c0 tx=0x7ffaf4019040 comp rx=0 tx=0).stop 2024-03-26T01:59:09.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:59:09.936+0000 7ffaeaffd700 1 -- 172.21.15.72:0/1877411221 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb041032b0 msgr2=0x7ffb04197db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:59:09.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:59:09.936+0000 7ffaeaffd700 1 --2- 172.21.15.72:0/1877411221 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb041032b0 0x7ffb04197db0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ffaf800ec40 tx=0x7ffaf800ef50 comp rx=0 tx=0).stop 2024-03-26T01:59:09.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:59:09.936+0000 7ffaeaffd700 1 -- 172.21.15.72:0/1877411221 shutdown_connections 2024-03-26T01:59:09.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:59:09.936+0000 7ffaeaffd700 1 --2- 172.21.15.72:0/1877411221 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ffb04103c50 0x7ffb0419ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:59:09.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:59:09.936+0000 7ffaeaffd700 1 --2- 172.21.15.72:0/1877411221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ffb041020b0 0x7ffb04197870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:59:09.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:59:09.936+0000 7ffaeaffd700 1 --2- 172.21.15.72:0/1877411221 >> v2:172.21.15.72:6800/2923394687 conn(0x7ffaec06d990 0x7ffaec06fe50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:59:09.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:59:09.936+0000 7ffaeaffd700 1 --2- 172.21.15.72:0/1877411221 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ffb041032b0 0x7ffb04197db0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:59:09.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:59:09.936+0000 7ffaeaffd700 1 -- 172.21.15.72:0/1877411221 >> 172.21.15.72:0/1877411221 conn(0x7ffb040fd690 msgr2=0x7ffb041066d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:59:09.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:59:09.937+0000 7ffaeaffd700 1 -- 172.21.15.72:0/1877411221 shutdown_connections 2024-03-26T01:59:09.937 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T01:59:09.937+0000 7ffaeaffd700 1 -- 172.21.15.72:0/1877411221 wait complete. 2024-03-26T01:59:10.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:10 smithi072 ceph-mon[30177]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:10.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:10 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:10.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:10 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:10.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:10 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:10.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:10 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:59:10.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:10 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:10.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:10 smithi072 ceph-mon[30177]: from='osd.3 v2:172.21.15.72:6813/2677886334' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T01:59:10.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:10 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:10.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:10 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:10.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:10 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:59:10.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:10 smithi072 ceph-mon[41619]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:10.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:10 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:10.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:10 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:10.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:10 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:10.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:10 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:59:10.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:10 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:10.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:10 smithi072 ceph-mon[41619]: from='osd.3 v2:172.21.15.72:6813/2677886334' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T01:59:10.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:10 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:10.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:10 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:10.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:10 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:59:10.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:10 smithi173 ceph-mon[33839]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:10.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:10 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:10.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:10 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:10.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:10 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:10.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:10 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:59:10.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:10 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:10.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:10 smithi173 ceph-mon[33839]: from='osd.3 v2:172.21.15.72:6813/2677886334' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T01:59:10.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:10 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:10.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:10 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:10.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:10 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:59:10.575 DEBUG:teuthology.orchestra.run.smithi072:osd.3> sudo journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.3.service 2024-03-26T01:59:10.578 INFO:tasks.cephadm:Deploying osd.4 on smithi173 with /dev/vg_nvme/lv_4... 2024-03-26T01:59:10.578 DEBUG:teuthology.orchestra.run.smithi173:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T01:59:10.822 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T01:59:11.246 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:11 smithi173 ceph-mon[33839]: from='osd.3 v2:172.21.15.72:6813/2677886334' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T01:59:11.246 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:11 smithi173 ceph-mon[33839]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T01:59:11.246 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:11 smithi173 ceph-mon[33839]: from='osd.3 v2:172.21.15.72:6813/2677886334' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-26T01:59:11.246 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:11 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:59:11.421 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:11 smithi072 ceph-mon[41619]: from='osd.3 v2:172.21.15.72:6813/2677886334' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T01:59:11.421 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:11 smithi072 ceph-mon[41619]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T01:59:11.421 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:11 smithi072 ceph-mon[41619]: from='osd.3 v2:172.21.15.72:6813/2677886334' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-26T01:59:11.422 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:11 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:59:11.422 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:11 smithi072 ceph-mon[30177]: from='osd.3 v2:172.21.15.72:6813/2677886334' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T01:59:11.422 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:11 smithi072 ceph-mon[30177]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T01:59:11.422 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:11 smithi072 ceph-mon[30177]: from='osd.3 v2:172.21.15.72:6813/2677886334' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]: dispatch 2024-03-26T01:59:11.422 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:11 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:59:11.701 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 26 01:59:11 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-3[65068]: 2024-03-26T01:59:11.589+0000 7fd1e3ee9700 -1 osd.3 0 waiting for initial osdmap 2024-03-26T01:59:11.701 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 26 01:59:11 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-3[65068]: 2024-03-26T01:59:11.592+0000 7fd1db4d3700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T01:59:12.256 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:12 smithi072 ceph-mon[30177]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:12.256 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:12 smithi072 ceph-mon[30177]: from='osd.3 v2:172.21.15.72:6813/2677886334' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-03-26T01:59:12.256 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:12 smithi072 ceph-mon[30177]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T01:59:12.257 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:12 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:59:12.257 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:12 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:59:12.257 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:12 smithi072 ceph-mon[30177]: from='osd.3 v2:172.21.15.72:6813/2677886334' entity='osd.3' 2024-03-26T01:59:12.257 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:12 smithi072 ceph-mon[41619]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:12.257 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:12 smithi072 ceph-mon[41619]: from='osd.3 v2:172.21.15.72:6813/2677886334' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-03-26T01:59:12.257 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:12 smithi072 ceph-mon[41619]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T01:59:12.257 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:59:12.257 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:12 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:59:12.257 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:12 smithi072 ceph-mon[41619]: from='osd.3 v2:172.21.15.72:6813/2677886334' entity='osd.3' 2024-03-26T01:59:12.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:12 smithi173 ceph-mon[33839]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:12.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:12 smithi173 ceph-mon[33839]: from='osd.3 v2:172.21.15.72:6813/2677886334' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi072", "root=default"]}]': finished 2024-03-26T01:59:12.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:12 smithi173 ceph-mon[33839]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T01:59:12.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:12 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:59:12.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:12 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:59:12.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:12 smithi173 ceph-mon[33839]: from='osd.3 v2:172.21.15.72:6813/2677886334' entity='osd.3' 2024-03-26T01:59:12.748 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T01:59:12.748 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T01:59:12.748 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-03-26T01:59:12.748 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-03-26T01:59:12.749 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128332 s, 817 MB/s 2024-03-26T01:59:12.749 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-03-26T01:59:13.349 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:13 smithi173 ceph-mon[33839]: purged_snaps scrub starts 2024-03-26T01:59:13.349 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:13 smithi173 ceph-mon[33839]: purged_snaps scrub ok 2024-03-26T01:59:13.349 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:13 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:59:13.349 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:13 smithi173 ceph-mon[33839]: osd.3 v2:172.21.15.72:6813/2677886334 boot 2024-03-26T01:59:13.349 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:13 smithi173 ceph-mon[33839]: osdmap e25: 4 total, 4 up, 4 in 2024-03-26T01:59:13.349 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:13 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:59:13.376 DEBUG:teuthology.orchestra.run.smithi173:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi173:vg_nvme/lv_4 2024-03-26T01:59:13.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:13 smithi072 ceph-mon[30177]: purged_snaps scrub starts 2024-03-26T01:59:13.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:13 smithi072 ceph-mon[30177]: purged_snaps scrub ok 2024-03-26T01:59:13.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:13 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:59:13.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:13 smithi072 ceph-mon[30177]: osd.3 v2:172.21.15.72:6813/2677886334 boot 2024-03-26T01:59:13.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:13 smithi072 ceph-mon[30177]: osdmap e25: 4 total, 4 up, 4 in 2024-03-26T01:59:13.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:13 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:59:13.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:13 smithi072 ceph-mon[41619]: purged_snaps scrub starts 2024-03-26T01:59:13.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:13 smithi072 ceph-mon[41619]: purged_snaps scrub ok 2024-03-26T01:59:13.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:13 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:59:13.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:13 smithi072 ceph-mon[41619]: osd.3 v2:172.21.15.72:6813/2677886334 boot 2024-03-26T01:59:13.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:13 smithi072 ceph-mon[41619]: osdmap e25: 4 total, 4 up, 4 in 2024-03-26T01:59:13.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:13 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T01:59:13.609 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T01:59:14.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:14 smithi173 ceph-mon[33839]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:14.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:14 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:14.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:14 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:14.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:14 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:59:14.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:14 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:59:14.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:14 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:59:14.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:14 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:59:14.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:14 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:14.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:14 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:14.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:14 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:59:14.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:14 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:14.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:14 smithi072 ceph-mon[30177]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:14.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:14 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:14.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:14 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:14.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:14 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:59:14.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:14 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:59:14.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:14 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:59:14.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:14 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:59:14.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:14 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:14.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:14 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:14.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:14 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:59:14.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:14 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:14.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:14 smithi072 ceph-mon[41619]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-26T01:59:14.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:14 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:14.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:14 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:14.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:14 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:59:14.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:14 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:59:14.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:14 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:59:14.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:14 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:59:14.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:14 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:14.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:14 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:14.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:14 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:59:14.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:14 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:15.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.205+0000 7fe31a831700 1 -- 172.21.15.173:0/2531807783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe314102b40 msgr2=0x7fe314104f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:59:15.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.205+0000 7fe31a831700 1 --2- 172.21.15.173:0/2531807783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe314102b40 0x7fe314104f30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe308009990 tx=0x7fe308009ca0 comp rx=0 tx=0).stop 2024-03-26T01:59:15.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.205+0000 7fe31a831700 1 -- 172.21.15.173:0/2531807783 shutdown_connections 2024-03-26T01:59:15.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.205+0000 7fe31a831700 1 --2- 172.21.15.173:0/2531807783 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe314107da0 0x7fe31410a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:59:15.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.205+0000 7fe31a831700 1 --2- 172.21.15.173:0/2531807783 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe314102b40 0x7fe314104f30 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:59:15.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.205+0000 7fe31a831700 1 --2- 172.21.15.173:0/2531807783 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe314105470 0x7fe314107860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:59:15.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.205+0000 7fe31a831700 1 -- 172.21.15.173:0/2531807783 >> 172.21.15.173:0/2531807783 conn(0x7fe3140fc570 msgr2=0x7fe3140fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:59:15.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.205+0000 7fe31a831700 1 -- 172.21.15.173:0/2531807783 shutdown_connections 2024-03-26T01:59:15.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.205+0000 7fe31a831700 1 -- 172.21.15.173:0/2531807783 wait complete. 2024-03-26T01:59:15.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.206+0000 7fe31a831700 1 Processor -- start 2024-03-26T01:59:15.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.206+0000 7fe31a831700 1 -- start start 2024-03-26T01:59:15.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.206+0000 7fe31a831700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe314102b40 0x7fe314197880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:59:15.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.206+0000 7fe31a831700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe314105470 0x7fe314197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:59:15.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.206+0000 7fe31a831700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe314107da0 0x7fe31419ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:59:15.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.206+0000 7fe31a831700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31419f410 con 0x7fe314102b40 2024-03-26T01:59:15.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.206+0000 7fe31a831700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe31419f550 con 0x7fe314107da0 2024-03-26T01:59:15.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.206+0000 7fe31a831700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe31419f6e0 con 0x7fe314105470 2024-03-26T01:59:15.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.206+0000 7fe318dce700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe314107da0 0x7fe31419ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:59:15.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.206+0000 7fe318dce700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe314107da0 0x7fe31419ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:37958/0 (socket says 172.21.15.173:37958) 2024-03-26T01:59:15.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.206+0000 7fe318dce700 1 -- 172.21.15.173:0/2214341820 learned_addr learned my addr 172.21.15.173:0/2214341820 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-26T01:59:15.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.206+0000 7fe3137fe700 1 --2- 172.21.15.173:0/2214341820 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe314105470 0x7fe314197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:59:15.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.206+0000 7fe313fff700 1 --2- 172.21.15.173:0/2214341820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe314102b40 0x7fe314197880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:59:15.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.207+0000 7fe318dce700 1 -- 172.21.15.173:0/2214341820 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe314105470 msgr2=0x7fe314197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:59:15.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.207+0000 7fe318dce700 1 --2- 172.21.15.173:0/2214341820 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe314105470 0x7fe314197dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:59:15.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.207+0000 7fe318dce700 1 -- 172.21.15.173:0/2214341820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe314102b40 msgr2=0x7fe314197880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:59:15.210 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.207+0000 7fe318dce700 1 --2- 172.21.15.173:0/2214341820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe314102b40 0x7fe314197880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:59:15.210 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.207+0000 7fe318dce700 1 -- 172.21.15.173:0/2214341820 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3080095f0 con 0x7fe314107da0 2024-03-26T01:59:15.210 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.207+0000 7fe313fff700 1 --2- 172.21.15.173:0/2214341820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe314102b40 0x7fe314197880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T01:59:15.210 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.207+0000 7fe318dce700 1 --2- 172.21.15.173:0/2214341820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe314107da0 0x7fe31419ee40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe30c00b9e0 tx=0x7fe30c00bcf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:59:15.210 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.207+0000 7fe3117fa700 1 -- 172.21.15.173:0/2214341820 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30c00c930 con 0x7fe314107da0 2024-03-26T01:59:15.210 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.207+0000 7fe31a831700 1 -- 172.21.15.173:0/2214341820 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe31419f9c0 con 0x7fe314107da0 2024-03-26T01:59:15.210 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.207+0000 7fe3117fa700 1 -- 172.21.15.173:0/2214341820 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fe30c014440 con 0x7fe314107da0 2024-03-26T01:59:15.210 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.207+0000 7fe3117fa700 1 -- 172.21.15.173:0/2214341820 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30c0125a0 con 0x7fe314107da0 2024-03-26T01:59:15.210 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.207+0000 7fe31a831700 1 -- 172.21.15.173:0/2214341820 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe31419ff40 con 0x7fe314107da0 2024-03-26T01:59:15.211 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.208+0000 7fe31a831700 1 -- 172.21.15.173:0/2214341820 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3140fe0f0 con 0x7fe314107da0 2024-03-26T01:59:15.211 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.208+0000 7fe3117fa700 1 -- 172.21.15.173:0/2214341820 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 15) v1 ==== 91363+0+0 (secure 0 0 0) 0x7fe30c00ca90 con 0x7fe314107da0 2024-03-26T01:59:15.211 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.209+0000 7fe3117fa700 1 --2- 172.21.15.173:0/2214341820 >> v2:172.21.15.72:6800/2923394687 conn(0x7fe2fc06d990 0x7fe2fc06fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:59:15.211 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.209+0000 7fe3117fa700 1 -- 172.21.15.173:0/2214341820 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 2803+0+0 (secure 0 0 0) 0x7fe30c006e40 con 0x7fe314107da0 2024-03-26T01:59:15.211 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.209+0000 7fe313fff700 1 --2- 172.21.15.173:0/2214341820 >> v2:172.21.15.72:6800/2923394687 conn(0x7fe2fc06d990 0x7fe2fc06fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:59:15.211 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.209+0000 7fe313fff700 1 --2- 172.21.15.173:0/2214341820 >> v2:172.21.15.72:6800/2923394687 conn(0x7fe2fc06d990 0x7fe2fc06fe50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe30800b5c0 tx=0x7fe308005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:59:15.214 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.214+0000 7fe3117fa700 1 -- 172.21.15.173:0/2214341820 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe30c056350 con 0x7fe314107da0 2024-03-26T01:59:15.312 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:15 smithi173 ceph-mon[33839]: Detected new or changed devices on smithi072 2024-03-26T01:59:15.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:15 smithi173 ceph-mon[33839]: Adjusting osd_memory_target on smithi072 to 4016M 2024-03-26T01:59:15.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:15 smithi173 ceph-mon[33839]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T01:59:15.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:15.415+0000 7fe31a831700 1 -- 172.21.15.173:0/2214341820 --> v2:172.21.15.72:6800/2923394687 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fe31410d4a0 con 0x7fe2fc06d990 2024-03-26T01:59:15.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:15 smithi072 ceph-mon[30177]: Detected new or changed devices on smithi072 2024-03-26T01:59:15.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:15 smithi072 ceph-mon[30177]: Adjusting osd_memory_target on smithi072 to 4016M 2024-03-26T01:59:15.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:15 smithi072 ceph-mon[30177]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T01:59:15.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:15 smithi072 ceph-mon[41619]: Detected new or changed devices on smithi072 2024-03-26T01:59:15.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:15 smithi072 ceph-mon[41619]: Adjusting osd_memory_target on smithi072 to 4016M 2024-03-26T01:59:15.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:15 smithi072 ceph-mon[41619]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T01:59:16.278 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:16 smithi173 ceph-mon[33839]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:16.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:16 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:59:16.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:16 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:59:16.279 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:16 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:16.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:16 smithi072 ceph-mon[30177]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:16.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:16 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:59:16.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:16 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:59:16.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:16 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:16.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:16 smithi072 ceph-mon[41619]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:16.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:16 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:59:16.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:16 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:59:16.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:16 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:17.260 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:17 smithi173 ceph-mon[33839]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:17.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:17 smithi072 ceph-mon[30177]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:17.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:17 smithi072 ceph-mon[41619]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:18.461 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:18 smithi173 ceph-mon[33839]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:18.461 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.173:0/3475100957' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b3e8749-94e2-4727-9d1e-075ef3e43efd"}]: dispatch 2024-03-26T01:59:18.461 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b3e8749-94e2-4727-9d1e-075ef3e43efd"}]: dispatch 2024-03-26T01:59:18.461 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5b3e8749-94e2-4727-9d1e-075ef3e43efd"}]': finished 2024-03-26T01:59:18.461 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:18 smithi173 ceph-mon[33839]: osdmap e27: 5 total, 4 up, 5 in 2024-03-26T01:59:18.461 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:18 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:59:18.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:18 smithi072 ceph-mon[30177]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:18.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.173:0/3475100957' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b3e8749-94e2-4727-9d1e-075ef3e43efd"}]: dispatch 2024-03-26T01:59:18.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b3e8749-94e2-4727-9d1e-075ef3e43efd"}]: dispatch 2024-03-26T01:59:18.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5b3e8749-94e2-4727-9d1e-075ef3e43efd"}]': finished 2024-03-26T01:59:18.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:18 smithi072 ceph-mon[30177]: osdmap e27: 5 total, 4 up, 5 in 2024-03-26T01:59:18.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:18 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:59:18.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:18 smithi072 ceph-mon[41619]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:18.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.173:0/3475100957' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b3e8749-94e2-4727-9d1e-075ef3e43efd"}]: dispatch 2024-03-26T01:59:18.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5b3e8749-94e2-4727-9d1e-075ef3e43efd"}]: dispatch 2024-03-26T01:59:18.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5b3e8749-94e2-4727-9d1e-075ef3e43efd"}]': finished 2024-03-26T01:59:18.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:18 smithi072 ceph-mon[41619]: osdmap e27: 5 total, 4 up, 5 in 2024-03-26T01:59:18.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:18 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:59:19.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.173:0/3808030338' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:59:19.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.173:0/3808030338' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:59:19.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.173:0/3808030338' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:59:20.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:20 smithi072 ceph-mon[41619]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:20.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:20 smithi072 ceph-mon[30177]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:20.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:20 smithi173 ceph-mon[33839]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:22.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:22 smithi072 ceph-mon[30177]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:22.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:22 smithi072 ceph-mon[41619]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:22.495 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:22 smithi173 ceph-mon[33839]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:24.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:24 smithi072 ceph-mon[30177]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:24.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:24 smithi072 ceph-mon[41619]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:24.472 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:24 smithi173 ceph-mon[33839]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:25.434 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:25 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T01:59:25.434 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:25 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:25.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:25 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T01:59:25.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:25 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:25.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:25 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T01:59:25.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:25 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:26.312 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:26 smithi173 ceph-mon[33839]: Deploying daemon osd.4 on smithi173 2024-03-26T01:59:26.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:26 smithi173 ceph-mon[33839]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:26.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:26 smithi072 ceph-mon[30177]: Deploying daemon osd.4 on smithi173 2024-03-26T01:59:26.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:26 smithi072 ceph-mon[30177]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:26.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:26 smithi072 ceph-mon[41619]: Deploying daemon osd.4 on smithi173 2024-03-26T01:59:26.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:26 smithi072 ceph-mon[41619]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:28.307 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:28 smithi173 ceph-mon[33839]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:28.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:28 smithi072 ceph-mon[30177]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:28.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:28 smithi072 ceph-mon[41619]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:30.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:30 smithi072 ceph-mon[30177]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:30.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:30 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:59:30.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:30 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:30.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:30 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:30.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:30 smithi072 ceph-mon[41619]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:30.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:30 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:59:30.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:30 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:30.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:30 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:30.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:30 smithi173 ceph-mon[33839]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:30.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:30 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:59:30.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:30 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:30.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:30 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:32.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:32 smithi173 ceph-mon[33839]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:32.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:32 smithi072 ceph-mon[30177]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:32.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:32 smithi072 ceph-mon[41619]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:33.681 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 4 on host 'smithi173' 2024-03-26T01:59:33.681 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:33.679+0000 7fe3117fa700 1 -- 172.21.15.173:0/2214341820 <== mgr.14152 v2:172.21.15.72:6800/2923394687 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe31410d4a0 con 0x7fe2fc06d990 2024-03-26T01:59:33.683 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:33.682+0000 7fe31a831700 1 -- 172.21.15.173:0/2214341820 >> v2:172.21.15.72:6800/2923394687 conn(0x7fe2fc06d990 msgr2=0x7fe2fc06fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:59:33.683 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:33.682+0000 7fe31a831700 1 --2- 172.21.15.173:0/2214341820 >> v2:172.21.15.72:6800/2923394687 conn(0x7fe2fc06d990 0x7fe2fc06fe50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe30800b5c0 tx=0x7fe308005fd0 comp rx=0 tx=0).stop 2024-03-26T01:59:33.683 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:33.682+0000 7fe31a831700 1 -- 172.21.15.173:0/2214341820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe314107da0 msgr2=0x7fe31419ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:59:33.683 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:33.682+0000 7fe31a831700 1 --2- 172.21.15.173:0/2214341820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe314107da0 0x7fe31419ee40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe30c00b9e0 tx=0x7fe30c00bcf0 comp rx=0 tx=0).stop 2024-03-26T01:59:33.683 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:33.682+0000 7fe31a831700 1 -- 172.21.15.173:0/2214341820 shutdown_connections 2024-03-26T01:59:33.683 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:33.682+0000 7fe31a831700 1 --2- 172.21.15.173:0/2214341820 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe314107da0 0x7fe31419ee40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:59:33.683 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:33.682+0000 7fe31a831700 1 --2- 172.21.15.173:0/2214341820 >> v2:172.21.15.72:6800/2923394687 conn(0x7fe2fc06d990 0x7fe2fc06fe50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:59:33.683 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:33.682+0000 7fe31a831700 1 --2- 172.21.15.173:0/2214341820 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe314102b40 0x7fe314197880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:59:33.683 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:33.682+0000 7fe31a831700 1 --2- 172.21.15.173:0/2214341820 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe314105470 0x7fe314197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:59:33.683 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:33.682+0000 7fe31a831700 1 -- 172.21.15.173:0/2214341820 >> 172.21.15.173:0/2214341820 conn(0x7fe3140fc570 msgr2=0x7fe3141037b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:59:33.683 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:33.682+0000 7fe31a831700 1 -- 172.21.15.173:0/2214341820 shutdown_connections 2024-03-26T01:59:33.683 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:33.682+0000 7fe31a831700 1 -- 172.21.15.173:0/2214341820 wait complete. 2024-03-26T01:59:34.044 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:33 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:34.045 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:33 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:34.045 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:33 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:34.045 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:33 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:59:34.045 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:33 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:34.045 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:33 smithi173 ceph-mon[33839]: from='osd.4 v2:172.21.15.173:6800/510505310' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T01:59:34.045 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:33 smithi173 ceph-mon[33839]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T01:59:34.045 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:33 smithi173 ceph-mon[33839]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:34.045 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:33 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:34.046 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:33 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:34.046 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:33 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:59:34.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:33 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:34.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:33 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:34.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:33 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:34.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:33 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:59:34.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:33 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:34.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:33 smithi072 ceph-mon[41619]: from='osd.4 v2:172.21.15.173:6800/510505310' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T01:59:34.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:33 smithi072 ceph-mon[41619]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T01:59:34.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:33 smithi072 ceph-mon[41619]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:34.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:33 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:34.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:33 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:34.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:33 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:59:34.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:33 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:34.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:33 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:34.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:33 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:34.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:33 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:59:34.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:33 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:34.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:33 smithi072 ceph-mon[30177]: from='osd.4 v2:172.21.15.173:6800/510505310' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T01:59:34.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:33 smithi072 ceph-mon[30177]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T01:59:34.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:33 smithi072 ceph-mon[30177]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:34.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:33 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:34.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:33 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:34.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:33 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:59:34.354 DEBUG:teuthology.orchestra.run.smithi173:osd.4> sudo journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.4.service 2024-03-26T01:59:34.357 INFO:tasks.cephadm:Deploying osd.5 on smithi173 with /dev/vg_nvme/lv_3... 2024-03-26T01:59:34.357 DEBUG:teuthology.orchestra.run.smithi173:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T01:59:34.899 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:34 smithi173 ceph-mon[33839]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T01:59:34.899 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:34 smithi173 ceph-mon[33839]: from='osd.4 v2:172.21.15.173:6800/510505310' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-26T01:59:34.900 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:34 smithi173 ceph-mon[33839]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T01:59:34.900 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:34 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:59:34.900 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:34 smithi173 ceph-mon[33839]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-26T01:59:34.900 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 01:59:34 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-4[39374]: 2024-03-26T01:59:34.818+0000 7fc2f6829700 -1 osd.4 0 waiting for initial osdmap 2024-03-26T01:59:34.901 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 01:59:34 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-4[39374]: 2024-03-26T01:59:34.821+0000 7fc2f261c700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T01:59:35.214 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T01:59:35.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:34 smithi072 ceph-mon[30177]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T01:59:35.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:34 smithi072 ceph-mon[30177]: from='osd.4 v2:172.21.15.173:6800/510505310' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-26T01:59:35.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:34 smithi072 ceph-mon[30177]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T01:59:35.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:34 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:59:35.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:34 smithi072 ceph-mon[30177]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-26T01:59:35.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:34 smithi072 ceph-mon[41619]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T01:59:35.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:34 smithi072 ceph-mon[41619]: from='osd.4 v2:172.21.15.173:6800/510505310' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-26T01:59:35.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:34 smithi072 ceph-mon[41619]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T01:59:35.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:34 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:59:35.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:34 smithi072 ceph-mon[41619]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-26T01:59:36.114 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:35 smithi173 ceph-mon[33839]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-26T01:59:36.115 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:35 smithi173 ceph-mon[33839]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T01:59:36.115 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:35 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:59:36.115 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:35 smithi173 ceph-mon[33839]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:36.115 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:35 smithi173 ceph-mon[33839]: osd.4 v2:172.21.15.173:6800/510505310 boot 2024-03-26T01:59:36.115 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:35 smithi173 ceph-mon[33839]: osdmap e30: 5 total, 5 up, 5 in 2024-03-26T01:59:36.115 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:35 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:59:36.218 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:35 smithi072 ceph-mon[30177]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-26T01:59:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:35 smithi072 ceph-mon[30177]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T01:59:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:35 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:59:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:35 smithi072 ceph-mon[30177]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:35 smithi072 ceph-mon[30177]: osd.4 v2:172.21.15.173:6800/510505310 boot 2024-03-26T01:59:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:35 smithi072 ceph-mon[30177]: osdmap e30: 5 total, 5 up, 5 in 2024-03-26T01:59:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:35 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:59:36.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:35 smithi072 ceph-mon[41619]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-26T01:59:36.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:35 smithi072 ceph-mon[41619]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T01:59:36.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:35 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:59:36.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:35 smithi072 ceph-mon[41619]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-03-26T01:59:36.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:35 smithi072 ceph-mon[41619]: osd.4 v2:172.21.15.173:6800/510505310 boot 2024-03-26T01:59:36.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:35 smithi072 ceph-mon[41619]: osdmap e30: 5 total, 5 up, 5 in 2024-03-26T01:59:36.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:35 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T01:59:37.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:36 smithi173 ceph-mon[33839]: purged_snaps scrub starts 2024-03-26T01:59:37.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:36 smithi173 ceph-mon[33839]: purged_snaps scrub ok 2024-03-26T01:59:37.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:36 smithi173 ceph-mon[33839]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T01:59:37.218 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:36 smithi072 ceph-mon[30177]: purged_snaps scrub starts 2024-03-26T01:59:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:36 smithi072 ceph-mon[30177]: purged_snaps scrub ok 2024-03-26T01:59:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:36 smithi072 ceph-mon[30177]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T01:59:37.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:36 smithi072 ceph-mon[41619]: purged_snaps scrub starts 2024-03-26T01:59:37.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:36 smithi072 ceph-mon[41619]: purged_snaps scrub ok 2024-03-26T01:59:37.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:36 smithi072 ceph-mon[41619]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T01:59:38.120 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:37 smithi072 ceph-mon[30177]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:59:38.120 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:37 smithi072 ceph-mon[30177]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T01:59:38.121 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:37 smithi072 ceph-mon[41619]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:59:38.121 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:37 smithi072 ceph-mon[41619]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T01:59:38.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:37 smithi173 ceph-mon[33839]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:59:38.228 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:37 smithi173 ceph-mon[33839]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T01:59:38.814 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T01:59:38.814 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T01:59:38.814 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-03-26T01:59:38.814 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-03-26T01:59:38.814 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112651 s, 931 MB/s 2024-03-26T01:59:38.814 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-03-26T01:59:39.408 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:39 smithi173 ceph-mon[33839]: Detected new or changed devices on smithi173 2024-03-26T01:59:39.408 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:39 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:39.408 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:39 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:39.409 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:39 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:59:39.409 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:39 smithi173 ceph-mon[33839]: Adjusting osd_memory_target on smithi173 to 17088M 2024-03-26T01:59:39.409 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:39 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:39.409 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:39 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:39.409 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:39 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:59:39.409 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:39 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:39.409 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:39 smithi173 ceph-mon[33839]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T01:59:39.409 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:39 smithi173 ceph-mon[33839]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:59:39.436 DEBUG:teuthology.orchestra.run.smithi173:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi173:vg_nvme/lv_3 2024-03-26T01:59:39.672 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T01:59:39.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:39 smithi072 ceph-mon[30177]: Detected new or changed devices on smithi173 2024-03-26T01:59:39.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:39 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:39.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:39 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:39.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:39 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:59:39.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:39 smithi072 ceph-mon[30177]: Adjusting osd_memory_target on smithi173 to 17088M 2024-03-26T01:59:39.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:39 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:39.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:39 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:39.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:39 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:59:39.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:39 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:39.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:39 smithi072 ceph-mon[30177]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T01:59:39.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:39 smithi072 ceph-mon[30177]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:59:39.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:39 smithi072 ceph-mon[41619]: Detected new or changed devices on smithi173 2024-03-26T01:59:39.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:39 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:39.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:39 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:39.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:39 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T01:59:39.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:39 smithi072 ceph-mon[41619]: Adjusting osd_memory_target on smithi173 to 17088M 2024-03-26T01:59:39.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:39 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:39.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:39 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:39.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:39 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T01:59:39.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:39 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:39.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:39 smithi072 ceph-mon[41619]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T01:59:39.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:39 smithi072 ceph-mon[41619]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:59:40.943 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:40 smithi173 ceph-mon[33839]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T01:59:41.177 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.174+0000 7fc49f5e1700 1 -- 172.21.15.173:0/4232009302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc498100150 msgr2=0x7fc4981005d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:59:41.177 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.174+0000 7fc49f5e1700 1 --2- 172.21.15.173:0/4232009302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc498100150 0x7fc4981005d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc488009990 tx=0x7fc488009ca0 comp rx=0 tx=0).stop 2024-03-26T01:59:41.177 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.175+0000 7fc49f5e1700 1 -- 172.21.15.173:0/4232009302 shutdown_connections 2024-03-26T01:59:41.177 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.175+0000 7fc49f5e1700 1 --2- 172.21.15.173:0/4232009302 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc498100b10 0x7fc49810a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:59:41.177 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.175+0000 7fc49f5e1700 1 --2- 172.21.15.173:0/4232009302 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc4981036d0 0x7fc498103af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:59:41.177 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.175+0000 7fc49f5e1700 1 --2- 172.21.15.173:0/4232009302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc498100150 0x7fc4981005d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:59:41.177 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.175+0000 7fc49f5e1700 1 -- 172.21.15.173:0/4232009302 >> 172.21.15.173:0/4232009302 conn(0x7fc4980fbd10 msgr2=0x7fc4980fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T01:59:41.177 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.175+0000 7fc49f5e1700 1 -- 172.21.15.173:0/4232009302 shutdown_connections 2024-03-26T01:59:41.177 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.175+0000 7fc49f5e1700 1 -- 172.21.15.173:0/4232009302 wait complete. 2024-03-26T01:59:41.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.175+0000 7fc49f5e1700 1 Processor -- start 2024-03-26T01:59:41.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.175+0000 7fc49f5e1700 1 -- start start 2024-03-26T01:59:41.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.175+0000 7fc49f5e1700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc498100150 0x7fc498102dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:59:41.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.175+0000 7fc49f5e1700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc498100b10 0x7fc498101420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:59:41.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.175+0000 7fc49f5e1700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc4981036d0 0x7fc498101960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:59:41.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.175+0000 7fc49f5e1700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4981020e0 con 0x7fc498100b10 2024-03-26T01:59:41.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.175+0000 7fc49f5e1700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc498102250 con 0x7fc498100150 2024-03-26T01:59:41.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.175+0000 7fc49f5e1700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4981a1810 con 0x7fc4981036d0 2024-03-26T01:59:41.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.175+0000 7fc49d37d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc498100150 0x7fc498102dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:59:41.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.175+0000 7fc49d37d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc498100150 0x7fc498102dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:55950/0 (socket says 172.21.15.173:55950) 2024-03-26T01:59:41.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.176+0000 7fc49d37d700 1 -- 172.21.15.173:0/1891611887 learned_addr learned my addr 172.21.15.173:0/1891611887 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-26T01:59:41.178 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.176+0000 7fc49cb7c700 1 --2- 172.21.15.173:0/1891611887 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc498100b10 0x7fc498101420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:59:41.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.176+0000 7fc49db7e700 1 --2- 172.21.15.173:0/1891611887 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc4981036d0 0x7fc498101960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:59:41.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.176+0000 7fc49d37d700 1 -- 172.21.15.173:0/1891611887 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc4981036d0 msgr2=0x7fc498101960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:59:41.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.176+0000 7fc49d37d700 1 --2- 172.21.15.173:0/1891611887 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc4981036d0 0x7fc498101960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:59:41.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.176+0000 7fc49d37d700 1 -- 172.21.15.173:0/1891611887 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc498100b10 msgr2=0x7fc498101420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T01:59:41.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.176+0000 7fc49d37d700 1 --2- 172.21.15.173:0/1891611887 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc498100b10 0x7fc498101420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T01:59:41.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.176+0000 7fc49d37d700 1 -- 172.21.15.173:0/1891611887 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4880095f0 con 0x7fc498100150 2024-03-26T01:59:41.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.176+0000 7fc49cb7c700 1 --2- 172.21.15.173:0/1891611887 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc498100b10 0x7fc498101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T01:59:41.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.176+0000 7fc49db7e700 1 --2- 172.21.15.173:0/1891611887 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc4981036d0 0x7fc498101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T01:59:41.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.176+0000 7fc49d37d700 1 --2- 172.21.15.173:0/1891611887 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc498100150 0x7fc498102dd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc49400c970 tx=0x7fc49400cd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:59:41.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.176+0000 7fc48e7fc700 1 -- 172.21.15.173:0/1891611887 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc494004220 con 0x7fc498100150 2024-03-26T01:59:41.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.176+0000 7fc49f5e1700 1 -- 172.21.15.173:0/1891611887 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4981a1af0 con 0x7fc498100150 2024-03-26T01:59:41.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.176+0000 7fc48e7fc700 1 -- 172.21.15.173:0/1891611887 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fc494007d50 con 0x7fc498100150 2024-03-26T01:59:41.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.176+0000 7fc48e7fc700 1 -- 172.21.15.173:0/1891611887 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc494003d30 con 0x7fc498100150 2024-03-26T01:59:41.179 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.176+0000 7fc49f5e1700 1 -- 172.21.15.173:0/1891611887 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4981a2040 con 0x7fc498100150 2024-03-26T01:59:41.180 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.177+0000 7fc49f5e1700 1 -- 172.21.15.173:0/1891611887 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc498108e50 con 0x7fc498100150 2024-03-26T01:59:41.180 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.178+0000 7fc48e7fc700 1 -- 172.21.15.173:0/1891611887 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 15) v1 ==== 91363+0+0 (secure 0 0 0) 0x7fc4940078a0 con 0x7fc498100150 2024-03-26T01:59:41.180 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.178+0000 7fc48e7fc700 1 --2- 172.21.15.173:0/1891611887 >> v2:172.21.15.72:6800/2923394687 conn(0x7fc48406d990 0x7fc48406fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T01:59:41.180 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.178+0000 7fc48e7fc700 1 -- 172.21.15.173:0/1891611887 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 3205+0+0 (secure 0 0 0) 0x7fc49408c230 con 0x7fc498100150 2024-03-26T01:59:41.180 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.178+0000 7fc49cb7c700 1 --2- 172.21.15.173:0/1891611887 >> v2:172.21.15.72:6800/2923394687 conn(0x7fc48406d990 0x7fc48406fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T01:59:41.180 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.179+0000 7fc49cb7c700 1 --2- 172.21.15.173:0/1891611887 >> v2:172.21.15.72:6800/2923394687 conn(0x7fc48406d990 0x7fc48406fe50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc48800b5c0 tx=0x7fc488005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T01:59:41.183 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.183+0000 7fc48e7fc700 1 -- 172.21.15.173:0/1891611887 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fc494056790 con 0x7fc498100150 2024-03-26T01:59:41.218 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:40 smithi072 ceph-mon[30177]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T01:59:41.218 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:40 smithi072 ceph-mon[41619]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T01:59:41.386 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T01:59:41.386+0000 7fc49f5e1700 1 -- 172.21.15.173:0/1891611887 --> v2:172.21.15.72:6800/2923394687 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fc49810d4b0 con 0x7fc48406d990 2024-03-26T01:59:42.011 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:41 smithi173 ceph-mon[33839]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:59:42.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:41 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:59:42.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:41 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:59:42.012 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:41 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:42.218 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:41 smithi072 ceph-mon[30177]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:59:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:41 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:59:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:41 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:59:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:41 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:42.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:41 smithi072 ceph-mon[41619]: pgmap v121: 1 pgs: 1 remapped+peering; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:59:42.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:41 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T01:59:42.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:41 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T01:59:42.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:41 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:43.008 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:42 smithi173 ceph-mon[33839]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:43.218 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:42 smithi072 ceph-mon[30177]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:43.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:42 smithi072 ceph-mon[41619]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T01:59:44.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:43 smithi173 ceph-mon[33839]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T01:59:44.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:43 smithi072 ceph-mon[30177]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T01:59:44.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:43 smithi072 ceph-mon[41619]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T01:59:44.812 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:44 smithi173 ceph-mon[33839]: from='client.? 172.21.15.173:0/525605670' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "652ab183-9759-426c-83cd-c61a676b06c8"}]: dispatch 2024-03-26T01:59:44.812 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:44 smithi173 ceph-mon[33839]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "652ab183-9759-426c-83cd-c61a676b06c8"}]: dispatch 2024-03-26T01:59:44.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:44 smithi173 ceph-mon[33839]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "652ab183-9759-426c-83cd-c61a676b06c8"}]': finished 2024-03-26T01:59:44.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:44 smithi173 ceph-mon[33839]: osdmap e35: 6 total, 5 up, 6 in 2024-03-26T01:59:44.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:44 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T01:59:45.218 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:44 smithi072 ceph-mon[41619]: from='client.? 172.21.15.173:0/525605670' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "652ab183-9759-426c-83cd-c61a676b06c8"}]: dispatch 2024-03-26T01:59:45.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:44 smithi072 ceph-mon[41619]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "652ab183-9759-426c-83cd-c61a676b06c8"}]: dispatch 2024-03-26T01:59:45.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:44 smithi072 ceph-mon[41619]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "652ab183-9759-426c-83cd-c61a676b06c8"}]': finished 2024-03-26T01:59:45.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:44 smithi072 ceph-mon[41619]: osdmap e35: 6 total, 5 up, 6 in 2024-03-26T01:59:45.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:44 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T01:59:45.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:44 smithi072 ceph-mon[30177]: from='client.? 172.21.15.173:0/525605670' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "652ab183-9759-426c-83cd-c61a676b06c8"}]: dispatch 2024-03-26T01:59:45.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:44 smithi072 ceph-mon[30177]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "652ab183-9759-426c-83cd-c61a676b06c8"}]: dispatch 2024-03-26T01:59:45.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:44 smithi072 ceph-mon[30177]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "652ab183-9759-426c-83cd-c61a676b06c8"}]': finished 2024-03-26T01:59:45.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:44 smithi072 ceph-mon[30177]: osdmap e35: 6 total, 5 up, 6 in 2024-03-26T01:59:45.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:44 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T01:59:46.062 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:45 smithi173 ceph-mon[33839]: from='client.? 172.21.15.173:0/2235115213' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:59:46.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:45 smithi173 ceph-mon[33839]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-26T01:59:46.218 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:45 smithi072 ceph-mon[30177]: from='client.? 172.21.15.173:0/2235115213' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:59:46.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:45 smithi072 ceph-mon[30177]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-26T01:59:46.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:45 smithi072 ceph-mon[41619]: from='client.? 172.21.15.173:0/2235115213' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T01:59:46.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:45 smithi072 ceph-mon[41619]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-26T01:59:48.445 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:48 smithi173 ceph-mon[33839]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T01:59:48.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:48 smithi072 ceph-mon[30177]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T01:59:48.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:48 smithi072 ceph-mon[41619]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T01:59:50.270 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:50 smithi173 ceph-mon[33839]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-26T01:59:50.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:50 smithi072 ceph-mon[30177]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-26T01:59:50.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:50 smithi072 ceph-mon[41619]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-26T01:59:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:51 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T01:59:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:51 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:51.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:51 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T01:59:51.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:51 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:51.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:51 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T01:59:51.482 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:51 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T01:59:52.218 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:52 smithi173 ceph-mon[33839]: Deploying daemon osd.5 on smithi173 2024-03-26T01:59:52.218 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:52 smithi173 ceph-mon[33839]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T01:59:52.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:52 smithi072 ceph-mon[30177]: Deploying daemon osd.5 on smithi173 2024-03-26T01:59:52.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:52 smithi072 ceph-mon[30177]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T01:59:52.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:52 smithi072 ceph-mon[41619]: Deploying daemon osd.5 on smithi173 2024-03-26T01:59:52.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:52 smithi072 ceph-mon[41619]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T01:59:54.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:54 smithi173 ceph-mon[33839]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:59:54.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:54 smithi072 ceph-mon[30177]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:59:54.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:54 smithi072 ceph-mon[41619]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:59:56.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:56 smithi072 ceph-mon[30177]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:59:56.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:56 smithi072 ceph-mon[41619]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:59:56.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:56 smithi173 ceph-mon[33839]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:59:57.288 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:57 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:57.288 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:57 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:57.288 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:57 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:59:57.288 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 01:59:57 smithi072 ceph-mon[30177]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:59:57.288 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:57 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:57.288 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:57 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:57.289 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:57 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:59:57.289 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 01:59:57 smithi072 ceph-mon[41619]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T01:59:57.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:57 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:57.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:57 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T01:59:57.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:57 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T01:59:57.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 01:59:57 smithi173 ceph-mon[33839]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:00:00.052 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:00.051+0000 7fc48e7fc700 1 -- 172.21.15.173:0/1891611887 <== mgr.14152 v2:172.21.15.72:6800/2923394687 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc49810d4b0 con 0x7fc48406d990 2024-03-26T02:00:00.052 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 5 on host 'smithi173' 2024-03-26T02:00:00.054 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:00.053+0000 7fc49f5e1700 1 -- 172.21.15.173:0/1891611887 >> v2:172.21.15.72:6800/2923394687 conn(0x7fc48406d990 msgr2=0x7fc48406fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:00:00.054 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:00.053+0000 7fc49f5e1700 1 --2- 172.21.15.173:0/1891611887 >> v2:172.21.15.72:6800/2923394687 conn(0x7fc48406d990 0x7fc48406fe50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc48800b5c0 tx=0x7fc488005fd0 comp rx=0 tx=0).stop 2024-03-26T02:00:00.054 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:00.053+0000 7fc49f5e1700 1 -- 172.21.15.173:0/1891611887 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc498100150 msgr2=0x7fc498102dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:00:00.054 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:00.053+0000 7fc49f5e1700 1 --2- 172.21.15.173:0/1891611887 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc498100150 0x7fc498102dd0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc49400c970 tx=0x7fc49400cd30 comp rx=0 tx=0).stop 2024-03-26T02:00:00.054 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:00.054+0000 7fc49f5e1700 1 -- 172.21.15.173:0/1891611887 shutdown_connections 2024-03-26T02:00:00.055 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:00.054+0000 7fc49f5e1700 1 --2- 172.21.15.173:0/1891611887 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fc4981036d0 0x7fc498101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:00.055 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:00.054+0000 7fc49f5e1700 1 --2- 172.21.15.173:0/1891611887 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc498100150 0x7fc498102dd0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:00.055 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:00.054+0000 7fc49f5e1700 1 --2- 172.21.15.173:0/1891611887 >> v2:172.21.15.72:6800/2923394687 conn(0x7fc48406d990 0x7fc48406fe50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:00.055 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:00.054+0000 7fc49f5e1700 1 --2- 172.21.15.173:0/1891611887 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fc498100b10 0x7fc498101420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:00.055 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:00.054+0000 7fc49f5e1700 1 -- 172.21.15.173:0/1891611887 >> 172.21.15.173:0/1891611887 conn(0x7fc4980fbd10 msgr2=0x7fc4980fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:00:00.055 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:00.054+0000 7fc49f5e1700 1 -- 172.21.15.173:0/1891611887 shutdown_connections 2024-03-26T02:00:00.055 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:00.054+0000 7fc49f5e1700 1 -- 172.21.15.173:0/1891611887 wait complete. 2024-03-26T02:00:00.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:00 smithi173 ceph-mon[33839]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:00:00.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:00 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:00.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:00 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:00.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:00 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:00.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:00 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:00:00.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:00 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:00.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:00 smithi173 ceph-mon[33839]: from='osd.5 v2:172.21.15.173:6804/1295375838' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:00:00.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:00 smithi173 ceph-mon[33839]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:00:00.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:00 smithi173 ceph-mon[33839]: overall HEALTH_OK 2024-03-26T02:00:00.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:00 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:00.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:00 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:00.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:00 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:00:00.688 DEBUG:teuthology.orchestra.run.smithi173:osd.5> sudo journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.5.service 2024-03-26T02:00:00.690 INFO:tasks.cephadm:Deploying osd.6 on smithi173 with /dev/vg_nvme/lv_2... 2024-03-26T02:00:00.691 DEBUG:teuthology.orchestra.run.smithi173:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T02:00:00.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:00 smithi072 ceph-mon[41619]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:00:00.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:00 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:00.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:00 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:00.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:00 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:00.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:00 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:00:00.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:00 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:00.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:00 smithi072 ceph-mon[41619]: from='osd.5 v2:172.21.15.173:6804/1295375838' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:00:00.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:00 smithi072 ceph-mon[41619]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:00:00.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:00 smithi072 ceph-mon[41619]: overall HEALTH_OK 2024-03-26T02:00:00.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:00 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:00.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:00 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:00.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:00 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:00:00.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:00 smithi072 ceph-mon[30177]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:00:00.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:00 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:00.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:00 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:00.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:00 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:00.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:00 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:00:00.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:00 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:00.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:00 smithi072 ceph-mon[30177]: from='osd.5 v2:172.21.15.173:6804/1295375838' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:00:00.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:00 smithi072 ceph-mon[30177]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T02:00:00.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:00 smithi072 ceph-mon[30177]: overall HEALTH_OK 2024-03-26T02:00:00.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:00 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:00.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:00 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:00.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:00 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:00:01.301 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:01 smithi173 ceph-mon[33839]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T02:00:01.301 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:01 smithi173 ceph-mon[33839]: from='osd.5 v2:172.21.15.173:6804/1295375838' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-26T02:00:01.301 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:01 smithi173 ceph-mon[33839]: osdmap e36: 6 total, 5 up, 6 in 2024-03-26T02:00:01.301 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:01 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:00:01.301 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:01 smithi173 ceph-mon[33839]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-26T02:00:01.302 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:01 smithi173 ceph-mon[33839]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:00:01.302 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:00:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:00:01.244+0000 7f1d7475a700 -1 osd.5 0 waiting for initial osdmap 2024-03-26T02:00:01.302 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:00:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:00:01.250+0000 7f1d7054d700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:00:01.526 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:00:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:01 smithi072 ceph-mon[30177]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T02:00:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:01 smithi072 ceph-mon[30177]: from='osd.5 v2:172.21.15.173:6804/1295375838' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-26T02:00:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:01 smithi072 ceph-mon[30177]: osdmap e36: 6 total, 5 up, 6 in 2024-03-26T02:00:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:01 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:00:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:01 smithi072 ceph-mon[30177]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-26T02:00:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:01 smithi072 ceph-mon[30177]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:00:01.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:01 smithi072 ceph-mon[41619]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T02:00:01.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:01 smithi072 ceph-mon[41619]: from='osd.5 v2:172.21.15.173:6804/1295375838' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-26T02:00:01.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:01 smithi072 ceph-mon[41619]: osdmap e36: 6 total, 5 up, 6 in 2024-03-26T02:00:01.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:01 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:00:01.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:01 smithi072 ceph-mon[41619]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-26T02:00:01.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:01 smithi072 ceph-mon[41619]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-03-26T02:00:02.462 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:02 smithi173 ceph-mon[33839]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-26T02:00:02.462 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:02 smithi173 ceph-mon[33839]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T02:00:02.463 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:02 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:00:02.463 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:02 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:00:02.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:02 smithi072 ceph-mon[41619]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-26T02:00:02.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:02 smithi072 ceph-mon[41619]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T02:00:02.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:02 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:00:02.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:02 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:00:02.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:02 smithi072 ceph-mon[30177]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-26T02:00:02.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:02 smithi072 ceph-mon[30177]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T02:00:02.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:02 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:00:02.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:02 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:00:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:03 smithi173 ceph-mon[33839]: purged_snaps scrub starts 2024-03-26T02:00:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:03 smithi173 ceph-mon[33839]: purged_snaps scrub ok 2024-03-26T02:00:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:03 smithi173 ceph-mon[33839]: osd.5 v2:172.21.15.173:6804/1295375838 boot 2024-03-26T02:00:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:03 smithi173 ceph-mon[33839]: osdmap e38: 6 total, 6 up, 6 in 2024-03-26T02:00:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:03 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:00:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:03 smithi173 ceph-mon[33839]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:03 smithi173 ceph-mon[33839]: osdmap e39: 6 total, 6 up, 6 in 2024-03-26T02:00:03.718 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:03 smithi072 ceph-mon[30177]: purged_snaps scrub starts 2024-03-26T02:00:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:03 smithi072 ceph-mon[30177]: purged_snaps scrub ok 2024-03-26T02:00:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:03 smithi072 ceph-mon[30177]: osd.5 v2:172.21.15.173:6804/1295375838 boot 2024-03-26T02:00:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:03 smithi072 ceph-mon[30177]: osdmap e38: 6 total, 6 up, 6 in 2024-03-26T02:00:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:03 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:00:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:03 smithi072 ceph-mon[30177]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:03 smithi072 ceph-mon[30177]: osdmap e39: 6 total, 6 up, 6 in 2024-03-26T02:00:03.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:03 smithi072 ceph-mon[41619]: purged_snaps scrub starts 2024-03-26T02:00:03.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:03 smithi072 ceph-mon[41619]: purged_snaps scrub ok 2024-03-26T02:00:03.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:03 smithi072 ceph-mon[41619]: osd.5 v2:172.21.15.173:6804/1295375838 boot 2024-03-26T02:00:03.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:03 smithi072 ceph-mon[41619]: osdmap e38: 6 total, 6 up, 6 in 2024-03-26T02:00:03.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:03 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:00:03.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:03 smithi072 ceph-mon[41619]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:03.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:03 smithi072 ceph-mon[41619]: osdmap e39: 6 total, 6 up, 6 in 2024-03-26T02:00:04.998 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T02:00:04.998 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T02:00:04.999 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-03-26T02:00:04.999 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-03-26T02:00:04.999 INFO:teuthology.orchestra.run.smithi173.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0143004 s, 733 MB/s 2024-03-26T02:00:04.999 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-03-26T02:00:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:05 smithi173 ceph-mon[33839]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T02:00:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:05 smithi173 ceph-mon[33839]: Detected new or changed devices on smithi173 2024-03-26T02:00:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:05 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:05 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:05 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:05 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:05 smithi173 ceph-mon[33839]: Adjusting osd_memory_target on smithi173 to 8544M 2024-03-26T02:00:05.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:05 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:05.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:05 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:05.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:05 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:00:05.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:05 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:05.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:05 smithi173 ceph-mon[33839]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:05.700 DEBUG:teuthology.orchestra.run.smithi173:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi173:vg_nvme/lv_2 2024-03-26T02:00:05.718 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:05 smithi072 ceph-mon[30177]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T02:00:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:05 smithi072 ceph-mon[30177]: Detected new or changed devices on smithi173 2024-03-26T02:00:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:05 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:05 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:05 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:05 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:05 smithi072 ceph-mon[30177]: Adjusting osd_memory_target on smithi173 to 8544M 2024-03-26T02:00:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:05 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:05 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:05 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:00:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:05 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:05 smithi072 ceph-mon[30177]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:05 smithi072 ceph-mon[41619]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T02:00:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:05 smithi072 ceph-mon[41619]: Detected new or changed devices on smithi173 2024-03-26T02:00:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:05 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:05 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:05 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:05 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:05 smithi072 ceph-mon[41619]: Adjusting osd_memory_target on smithi173 to 8544M 2024-03-26T02:00:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:05 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:05 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:05 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:00:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:05 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:05 smithi072 ceph-mon[41619]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:05.935 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:00:06.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:06 smithi072 ceph-mon[30177]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T02:00:06.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:06 smithi072 ceph-mon[41619]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T02:00:06.746 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:06 smithi173 ceph-mon[33839]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T02:00:07.532 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.530+0000 7faee891e700 1 -- 172.21.15.173:0/3988188037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faee41025b0 msgr2=0x7faee41029d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:00:07.532 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.530+0000 7faee891e700 1 --2- 172.21.15.173:0/3988188037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faee41025b0 0x7faee41029d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7faed40099c0 tx=0x7faed4009cd0 comp rx=0 tx=0).stop 2024-03-26T02:00:07.532 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.531+0000 7faee891e700 1 -- 172.21.15.173:0/3988188037 shutdown_connections 2024-03-26T02:00:07.532 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.531+0000 7faee891e700 1 --2- 172.21.15.173:0/3988188037 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faee4101b60 0x7faee4109420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:07.532 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.531+0000 7faee891e700 1 --2- 172.21.15.173:0/3988188037 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faee41025b0 0x7faee41029d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:07.532 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.531+0000 7faee891e700 1 --2- 172.21.15.173:0/3988188037 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faee41011c0 0x7faee4101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:07.532 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.531+0000 7faee891e700 1 -- 172.21.15.173:0/3988188037 >> 172.21.15.173:0/3988188037 conn(0x7faee40fbcd0 msgr2=0x7faee40fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:00:07.532 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.531+0000 7faee891e700 1 -- 172.21.15.173:0/3988188037 shutdown_connections 2024-03-26T02:00:07.532 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.531+0000 7faee891e700 1 -- 172.21.15.173:0/3988188037 wait complete. 2024-03-26T02:00:07.533 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:07 smithi173 ceph-mon[33839]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T02:00:07.533 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:07 smithi173 ceph-mon[33839]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:07.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.531+0000 7faee891e700 1 Processor -- start 2024-03-26T02:00:07.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.531+0000 7faee891e700 1 -- start start 2024-03-26T02:00:07.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.532+0000 7faee891e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faee41011c0 0x7faee4197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:00:07.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.532+0000 7faee891e700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faee4101b60 0x7faee4197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:00:07.535 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.532+0000 7faee891e700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faee41025b0 0x7faee419ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:00:07.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.532+0000 7faee891e700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee419f290 con 0x7faee41025b0 2024-03-26T02:00:07.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.532+0000 7faee891e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee419f3d0 con 0x7faee41011c0 2024-03-26T02:00:07.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.532+0000 7faee259c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faee41011c0 0x7faee4197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:00:07.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.532+0000 7faee891e700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee419f560 con 0x7faee4101b60 2024-03-26T02:00:07.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.532+0000 7faee259c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faee41011c0 0x7faee4197700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:52600/0 (socket says 172.21.15.173:52600) 2024-03-26T02:00:07.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.532+0000 7faee259c700 1 -- 172.21.15.173:0/331679746 learned_addr learned my addr 172.21.15.173:0/331679746 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-26T02:00:07.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.532+0000 7faee1d9b700 1 --2- 172.21.15.173:0/331679746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faee4101b60 0x7faee4197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:00:07.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.532+0000 7faee2d9d700 1 --2- 172.21.15.173:0/331679746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faee41025b0 0x7faee419ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:00:07.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.532+0000 7faee259c700 1 -- 172.21.15.173:0/331679746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faee4101b60 msgr2=0x7faee4197c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:00:07.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.532+0000 7faee259c700 1 --2- 172.21.15.173:0/331679746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faee4101b60 0x7faee4197c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:07.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.532+0000 7faee259c700 1 -- 172.21.15.173:0/331679746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faee41025b0 msgr2=0x7faee419ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:00:07.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.532+0000 7faee259c700 1 --2- 172.21.15.173:0/331679746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faee41025b0 0x7faee419ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:07.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.532+0000 7faee259c700 1 -- 172.21.15.173:0/331679746 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed40095f0 con 0x7faee41011c0 2024-03-26T02:00:07.536 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.532+0000 7faee2d9d700 1 --2- 172.21.15.173:0/331679746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faee41025b0 0x7faee419ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T02:00:07.537 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.532+0000 7faee259c700 1 --2- 172.21.15.173:0/331679746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faee41011c0 0x7faee4197700 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7faed40098e0 tx=0x7faed4009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:00:07.537 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.532+0000 7faed37fe700 1 -- 172.21.15.173:0/331679746 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed401d070 con 0x7faee41011c0 2024-03-26T02:00:07.537 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.533+0000 7faed37fe700 1 -- 172.21.15.173:0/331679746 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7faed4003f60 con 0x7faee41011c0 2024-03-26T02:00:07.537 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.533+0000 7faee891e700 1 -- 172.21.15.173:0/331679746 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faee419f7e0 con 0x7faee41011c0 2024-03-26T02:00:07.537 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.533+0000 7faed37fe700 1 -- 172.21.15.173:0/331679746 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed4022c00 con 0x7faee41011c0 2024-03-26T02:00:07.537 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.533+0000 7faee891e700 1 -- 172.21.15.173:0/331679746 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faee419fbf0 con 0x7faee41011c0 2024-03-26T02:00:07.537 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.534+0000 7faee891e700 1 -- 172.21.15.173:0/331679746 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee4068d40 con 0x7faee41011c0 2024-03-26T02:00:07.537 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.534+0000 7faed37fe700 1 -- 172.21.15.173:0/331679746 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 15) v1 ==== 91363+0+0 (secure 0 0 0) 0x7faed4022470 con 0x7faee41011c0 2024-03-26T02:00:07.537 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.535+0000 7faed37fe700 1 --2- 172.21.15.173:0/331679746 >> v2:172.21.15.72:6800/2923394687 conn(0x7faecc06d990 0x7faecc06fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:00:07.537 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.535+0000 7faed37fe700 1 -- 172.21.15.173:0/331679746 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 3497+0+0 (secure 0 0 0) 0x7faed408dca0 con 0x7faee41011c0 2024-03-26T02:00:07.537 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.535+0000 7faee1d9b700 1 --2- 172.21.15.173:0/331679746 >> v2:172.21.15.72:6800/2923394687 conn(0x7faecc06d990 0x7faecc06fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:00:07.538 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.535+0000 7faee1d9b700 1 --2- 172.21.15.173:0/331679746 >> v2:172.21.15.72:6800/2923394687 conn(0x7faecc06d990 0x7faecc06fe50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7faed8005950 tx=0x7faed800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:00:07.540 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.539+0000 7faed37fe700 1 -- 172.21.15.173:0/331679746 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7faed4058180 con 0x7faee41011c0 2024-03-26T02:00:07.718 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:07 smithi072 ceph-mon[30177]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T02:00:07.718 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:07 smithi072 ceph-mon[30177]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:07.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:07 smithi072 ceph-mon[41619]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T02:00:07.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:07 smithi072 ceph-mon[41619]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:07.742 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:07.742+0000 7faee891e700 1 -- 172.21.15.173:0/331679746 --> v2:172.21.15.72:6800/2923394687 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7faee40fd8b0 con 0x7faecc06d990 2024-03-26T02:00:08.620 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:08 smithi072 ceph-mon[30177]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:00:08.620 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:08 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:00:08.620 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:08 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:00:08.620 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:08 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:08.620 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:08 smithi072 ceph-mon[41619]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:00:08.621 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:08 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:00:08.621 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:08 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:00:08.621 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:08 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:08.677 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:08 smithi173 ceph-mon[33839]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:00:08.677 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:08 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:00:08.677 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:08 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:00:08.677 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:08 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:09.604 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:09 smithi173 ceph-mon[33839]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:09 smithi072 ceph-mon[30177]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:09.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:09 smithi072 ceph-mon[41619]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:10.810 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.173:0/388126683' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d77483d-4aec-4685-8048-ca58d3482de3"}]: dispatch 2024-03-26T02:00:10.810 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d77483d-4aec-4685-8048-ca58d3482de3"}]: dispatch 2024-03-26T02:00:10.810 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d77483d-4aec-4685-8048-ca58d3482de3"}]': finished 2024-03-26T02:00:10.810 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:10 smithi173 ceph-mon[33839]: osdmap e43: 7 total, 6 up, 7 in 2024-03-26T02:00:10.810 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:10 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:00:10.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.173:0/388126683' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d77483d-4aec-4685-8048-ca58d3482de3"}]: dispatch 2024-03-26T02:00:10.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d77483d-4aec-4685-8048-ca58d3482de3"}]: dispatch 2024-03-26T02:00:10.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d77483d-4aec-4685-8048-ca58d3482de3"}]': finished 2024-03-26T02:00:10.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:10 smithi072 ceph-mon[30177]: osdmap e43: 7 total, 6 up, 7 in 2024-03-26T02:00:10.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:10 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:00:10.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.173:0/388126683' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d77483d-4aec-4685-8048-ca58d3482de3"}]: dispatch 2024-03-26T02:00:10.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d77483d-4aec-4685-8048-ca58d3482de3"}]: dispatch 2024-03-26T02:00:10.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d77483d-4aec-4685-8048-ca58d3482de3"}]': finished 2024-03-26T02:00:10.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:10 smithi072 ceph-mon[41619]: osdmap e43: 7 total, 6 up, 7 in 2024-03-26T02:00:10.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:10 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:00:11.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:11 smithi173 ceph-mon[33839]: from='client.? 172.21.15.173:0/2594261233' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:00:11.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:11 smithi173 ceph-mon[33839]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:11.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:11 smithi072 ceph-mon[30177]: from='client.? 172.21.15.173:0/2594261233' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:00:11.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:11 smithi072 ceph-mon[30177]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:11.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:11 smithi072 ceph-mon[41619]: from='client.? 172.21.15.173:0/2594261233' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:00:11.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:11 smithi072 ceph-mon[41619]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:14.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:14 smithi072 ceph-mon[30177]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:14.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:14 smithi072 ceph-mon[41619]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:14.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:14 smithi173 ceph-mon[33839]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:16.257 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:16 smithi173 ceph-mon[33839]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:16.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:16 smithi072 ceph-mon[30177]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:16.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:16 smithi072 ceph-mon[41619]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:17.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:17 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T02:00:17.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:17 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:17.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:17 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T02:00:17.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:17 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:17.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:17 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T02:00:17.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:17 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:18.424 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:18 smithi173 ceph-mon[33839]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:18.425 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:18 smithi173 ceph-mon[33839]: Deploying daemon osd.6 on smithi173 2024-03-26T02:00:18.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:18 smithi072 ceph-mon[30177]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:18.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:18 smithi072 ceph-mon[30177]: Deploying daemon osd.6 on smithi173 2024-03-26T02:00:18.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:18 smithi072 ceph-mon[41619]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:18.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:18 smithi072 ceph-mon[41619]: Deploying daemon osd.6 on smithi173 2024-03-26T02:00:20.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:20 smithi072 ceph-mon[30177]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:20.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:20 smithi072 ceph-mon[41619]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:20.484 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:20 smithi173 ceph-mon[33839]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:22.447 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:22 smithi173 ceph-mon[33839]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:22.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:22 smithi072 ceph-mon[30177]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:22.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:22 smithi072 ceph-mon[41619]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:24.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:23 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:24.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:23 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:00:24.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:23 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:24.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:23 smithi173 ceph-mon[33839]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:24.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:23 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:24.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:23 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:00:24.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:23 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:24.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:23 smithi072 ceph-mon[30177]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:24.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:23 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:24.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:23 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:00:24.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:23 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:24.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:23 smithi072 ceph-mon[41619]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:26.401 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:26 smithi173 ceph-mon[33839]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:26.401 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:26 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:26.401 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:26 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:26.401 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:26 smithi173 ceph-mon[33839]: from='osd.6 v2:172.21.15.173:6808/3732303478' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:00:26.401 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:26 smithi173 ceph-mon[33839]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:00:26.457 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 6 on host 'smithi173' 2024-03-26T02:00:26.457 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:26.456+0000 7faed37fe700 1 -- 172.21.15.173:0/331679746 <== mgr.14152 v2:172.21.15.72:6800/2923394687 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7faee40fd8b0 con 0x7faecc06d990 2024-03-26T02:00:26.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:26.458+0000 7faee891e700 1 -- 172.21.15.173:0/331679746 >> v2:172.21.15.72:6800/2923394687 conn(0x7faecc06d990 msgr2=0x7faecc06fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:00:26.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:26.458+0000 7faee891e700 1 --2- 172.21.15.173:0/331679746 >> v2:172.21.15.72:6800/2923394687 conn(0x7faecc06d990 0x7faecc06fe50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7faed8005950 tx=0x7faed800b410 comp rx=0 tx=0).stop 2024-03-26T02:00:26.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:26.458+0000 7faee891e700 1 -- 172.21.15.173:0/331679746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faee41011c0 msgr2=0x7faee4197700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:00:26.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:26.458+0000 7faee891e700 1 --2- 172.21.15.173:0/331679746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faee41011c0 0x7faee4197700 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7faed40098e0 tx=0x7faed4009580 comp rx=0 tx=0).stop 2024-03-26T02:00:26.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:26.459+0000 7faee891e700 1 -- 172.21.15.173:0/331679746 shutdown_connections 2024-03-26T02:00:26.460 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:26.459+0000 7faee891e700 1 --2- 172.21.15.173:0/331679746 >> v2:172.21.15.72:6800/2923394687 conn(0x7faecc06d990 0x7faecc06fe50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:26.460 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:26.459+0000 7faee891e700 1 --2- 172.21.15.173:0/331679746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faee41025b0 0x7faee419ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:26.460 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:26.459+0000 7faee891e700 1 --2- 172.21.15.173:0/331679746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faee41011c0 0x7faee4197700 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:26.460 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:26.459+0000 7faee891e700 1 --2- 172.21.15.173:0/331679746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faee4101b60 0x7faee4197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:26.460 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:26.459+0000 7faee891e700 1 -- 172.21.15.173:0/331679746 >> 172.21.15.173:0/331679746 conn(0x7faee40fbcd0 msgr2=0x7faee40fd1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:00:26.460 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:26.459+0000 7faee891e700 1 -- 172.21.15.173:0/331679746 shutdown_connections 2024-03-26T02:00:26.460 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:26.459+0000 7faee891e700 1 -- 172.21.15.173:0/331679746 wait complete. 2024-03-26T02:00:26.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:26 smithi072 ceph-mon[30177]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:26.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:26 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:26.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:26 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:26.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:26 smithi072 ceph-mon[30177]: from='osd.6 v2:172.21.15.173:6808/3732303478' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:00:26.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:26 smithi072 ceph-mon[30177]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:00:26.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:26 smithi072 ceph-mon[41619]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:26.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:26 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:26.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:26 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:26.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:26 smithi072 ceph-mon[41619]: from='osd.6 v2:172.21.15.173:6808/3732303478' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:00:26.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:26 smithi072 ceph-mon[41619]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T02:00:27.155 DEBUG:teuthology.orchestra.run.smithi173:osd.6> sudo journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.6.service 2024-03-26T02:00:27.157 INFO:tasks.cephadm:Deploying osd.7 on smithi173 with /dev/vg_nvme/lv_1... 2024-03-26T02:00:27.158 DEBUG:teuthology.orchestra.run.smithi173:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T02:00:27.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:27 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:27.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:27 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:27.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:27 smithi072 ceph-mon[30177]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T02:00:27.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:27 smithi072 ceph-mon[30177]: from='osd.6 v2:172.21.15.173:6808/3732303478' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-26T02:00:27.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:27 smithi072 ceph-mon[30177]: osdmap e44: 7 total, 6 up, 7 in 2024-03-26T02:00:27.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:27 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:00:27.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:27 smithi072 ceph-mon[30177]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-26T02:00:27.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:27 smithi072 ceph-mon[30177]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:27.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:27 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:27.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:27 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:27.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:27 smithi072 ceph-mon[41619]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T02:00:27.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:27 smithi072 ceph-mon[41619]: from='osd.6 v2:172.21.15.173:6808/3732303478' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-26T02:00:27.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:27 smithi072 ceph-mon[41619]: osdmap e44: 7 total, 6 up, 7 in 2024-03-26T02:00:27.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:27 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:00:27.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:27 smithi072 ceph-mon[41619]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-26T02:00:27.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:27 smithi072 ceph-mon[41619]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:27.749 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:27 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:27.749 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:27 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:27.750 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:27 smithi173 ceph-mon[33839]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T02:00:27.750 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:27 smithi173 ceph-mon[33839]: from='osd.6 v2:172.21.15.173:6808/3732303478' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-26T02:00:27.750 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:27 smithi173 ceph-mon[33839]: osdmap e44: 7 total, 6 up, 7 in 2024-03-26T02:00:27.750 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:27 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:00:27.750 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:27 smithi173 ceph-mon[33839]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-26T02:00:27.750 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:27 smithi173 ceph-mon[33839]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-03-26T02:00:27.751 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:00:27 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:00:27.586+0000 7f2df7069700 -1 osd.6 0 waiting for initial osdmap 2024-03-26T02:00:27.751 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:00:27 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:00:27.593+0000 7f2df2e5c700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:00:28.324 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:00:28.864 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:28 smithi173 ceph-mon[33839]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-26T02:00:28.865 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:28 smithi173 ceph-mon[33839]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T02:00:28.865 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:28 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:00:28.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:28 smithi072 ceph-mon[30177]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-26T02:00:28.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:28 smithi072 ceph-mon[30177]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T02:00:28.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:28 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:00:28.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:28 smithi072 ceph-mon[41619]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-26T02:00:28.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:28 smithi072 ceph-mon[41619]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T02:00:28.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:28 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:00:29.878 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:29 smithi173 ceph-mon[33839]: purged_snaps scrub starts 2024-03-26T02:00:29.878 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:29 smithi173 ceph-mon[33839]: purged_snaps scrub ok 2024-03-26T02:00:29.878 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:29 smithi173 ceph-mon[33839]: osd.6 v2:172.21.15.173:6808/3732303478 boot 2024-03-26T02:00:29.879 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:29 smithi173 ceph-mon[33839]: osdmap e46: 7 total, 7 up, 7 in 2024-03-26T02:00:29.879 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:29 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:00:29.879 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:29 smithi173 ceph-mon[33839]: osdmap e47: 7 total, 7 up, 7 in 2024-03-26T02:00:29.879 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:29 smithi173 ceph-mon[33839]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:29.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:29 smithi072 ceph-mon[41619]: purged_snaps scrub starts 2024-03-26T02:00:29.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:29 smithi072 ceph-mon[41619]: purged_snaps scrub ok 2024-03-26T02:00:29.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:29 smithi072 ceph-mon[41619]: osd.6 v2:172.21.15.173:6808/3732303478 boot 2024-03-26T02:00:29.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:29 smithi072 ceph-mon[41619]: osdmap e46: 7 total, 7 up, 7 in 2024-03-26T02:00:29.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:29 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:00:29.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:29 smithi072 ceph-mon[41619]: osdmap e47: 7 total, 7 up, 7 in 2024-03-26T02:00:29.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:29 smithi072 ceph-mon[41619]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:29.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:29 smithi072 ceph-mon[30177]: purged_snaps scrub starts 2024-03-26T02:00:29.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:29 smithi072 ceph-mon[30177]: purged_snaps scrub ok 2024-03-26T02:00:29.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:29 smithi072 ceph-mon[30177]: osd.6 v2:172.21.15.173:6808/3732303478 boot 2024-03-26T02:00:29.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:29 smithi072 ceph-mon[30177]: osdmap e46: 7 total, 7 up, 7 in 2024-03-26T02:00:29.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:29 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:00:29.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:29 smithi072 ceph-mon[30177]: osdmap e47: 7 total, 7 up, 7 in 2024-03-26T02:00:29.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:29 smithi072 ceph-mon[30177]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:30.780 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T02:00:30.781 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T02:00:30.781 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-03-26T02:00:30.781 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-03-26T02:00:30.781 INFO:teuthology.orchestra.run.smithi173.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0122395 s, 857 MB/s 2024-03-26T02:00:30.781 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-03-26T02:00:30.805 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:30 smithi173 ceph-mon[33839]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T02:00:31.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:30 smithi072 ceph-mon[30177]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T02:00:31.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:30 smithi072 ceph-mon[41619]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T02:00:31.959 DEBUG:teuthology.orchestra.run.smithi173:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi173:vg_nvme/lv_1 2024-03-26T02:00:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:31 smithi173 ceph-mon[33839]: osdmap e49: 7 total, 7 up, 7 in 2024-03-26T02:00:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:31 smithi173 ceph-mon[33839]: pgmap v161: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 163 KiB/s, 0 objects/s recovering 2024-03-26T02:00:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:31 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:31 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:31 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:32.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:31 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:32.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:31 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:32.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:31 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:32.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:31 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:32.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:31 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:00:32.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:31 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:32.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:31 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:00:32.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:31 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:32.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:31 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:00:32.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:31 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:32.197 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:00:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[30177]: osdmap e49: 7 total, 7 up, 7 in 2024-03-26T02:00:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[30177]: pgmap v161: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 163 KiB/s, 0 objects/s recovering 2024-03-26T02:00:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:00:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:00:32.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:32.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:00:32.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:32.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[41619]: osdmap e49: 7 total, 7 up, 7 in 2024-03-26T02:00:32.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[41619]: pgmap v161: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 163 KiB/s, 0 objects/s recovering 2024-03-26T02:00:32.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:32.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:32.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:32.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:32.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:32.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:32.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:32.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:00:32.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:32.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:00:32.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:32.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:00:32.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:31 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:33.062 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:32 smithi173 ceph-mon[33839]: Detected new or changed devices on smithi173 2024-03-26T02:00:33.062 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:32 smithi173 ceph-mon[33839]: Adjusting osd_memory_target on smithi173 to 5696M 2024-03-26T02:00:33.062 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:32 smithi173 ceph-mon[33839]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T02:00:33.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:32 smithi072 ceph-mon[30177]: Detected new or changed devices on smithi173 2024-03-26T02:00:33.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:32 smithi072 ceph-mon[30177]: Adjusting osd_memory_target on smithi173 to 5696M 2024-03-26T02:00:33.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:32 smithi072 ceph-mon[30177]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T02:00:33.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:32 smithi072 ceph-mon[41619]: Detected new or changed devices on smithi173 2024-03-26T02:00:33.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:32 smithi072 ceph-mon[41619]: Adjusting osd_memory_target on smithi173 to 5696M 2024-03-26T02:00:33.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:32 smithi072 ceph-mon[41619]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T02:00:33.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.821+0000 7f4418bc1700 1 -- 172.21.15.173:0/185338315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4414102b50 msgr2=0x7f4414109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:00:33.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.821+0000 7f4418bc1700 1 --2- 172.21.15.173:0/185338315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4414102b50 0x7f4414109400 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f440c00b0d0 tx=0x7f440c00b3e0 comp rx=0 tx=0).stop 2024-03-26T02:00:33.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.821+0000 7f4418bc1700 1 -- 172.21.15.173:0/185338315 shutdown_connections 2024-03-26T02:00:33.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.821+0000 7f4418bc1700 1 --2- 172.21.15.173:0/185338315 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4414102b50 0x7f4414109400 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:33.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.821+0000 7f4418bc1700 1 --2- 172.21.15.173:0/185338315 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4414100fb0 0x7f44141013d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:33.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.821+0000 7f4418bc1700 1 --2- 172.21.15.173:0/185338315 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f44141021b0 0x7f4414102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:33.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.821+0000 7f4418bc1700 1 -- 172.21.15.173:0/185338315 >> 172.21.15.173:0/185338315 conn(0x7f44140fc570 msgr2=0x7f44140fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:00:33.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.821+0000 7f4418bc1700 1 -- 172.21.15.173:0/185338315 shutdown_connections 2024-03-26T02:00:33.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.821+0000 7f4418bc1700 1 -- 172.21.15.173:0/185338315 wait complete. 2024-03-26T02:00:33.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.822+0000 7f4418bc1700 1 Processor -- start 2024-03-26T02:00:33.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.822+0000 7f4418bc1700 1 -- start start 2024-03-26T02:00:33.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.822+0000 7f4418bc1700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4414100fb0 0x7f441406b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:00:33.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.822+0000 7f4418bc1700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44141021b0 0x7f441419c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:00:33.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.822+0000 7f4418bc1700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4414102b50 0x7f441419eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:00:33.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.822+0000 7f4418bc1700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441419f240 con 0x7f4414100fb0 2024-03-26T02:00:33.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.822+0000 7f4418bc1700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f441419f3b0 con 0x7f44141021b0 2024-03-26T02:00:33.825 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.822+0000 7f4418bc1700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f441419f570 con 0x7f4414102b50 2024-03-26T02:00:33.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.822+0000 7f4411d9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44141021b0 0x7f441419c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:00:33.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.822+0000 7f4411d9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44141021b0 0x7f441419c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:43848/0 (socket says 172.21.15.173:43848) 2024-03-26T02:00:33.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.822+0000 7f4411d9b700 1 -- 172.21.15.173:0/3574714901 learned_addr learned my addr 172.21.15.173:0/3574714901 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-26T02:00:33.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.822+0000 7f441259c700 1 --2- 172.21.15.173:0/3574714901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4414100fb0 0x7f441406b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:00:33.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.823+0000 7f4412d9d700 1 --2- 172.21.15.173:0/3574714901 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4414102b50 0x7f441419eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:00:33.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.823+0000 7f4411d9b700 1 -- 172.21.15.173:0/3574714901 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4414102b50 msgr2=0x7f441419eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:00:33.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.823+0000 7f4411d9b700 1 --2- 172.21.15.173:0/3574714901 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4414102b50 0x7f441419eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:33.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.823+0000 7f4411d9b700 1 -- 172.21.15.173:0/3574714901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4414100fb0 msgr2=0x7f441406b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:00:33.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.823+0000 7f4411d9b700 1 --2- 172.21.15.173:0/3574714901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4414100fb0 0x7f441406b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:33.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.823+0000 7f4411d9b700 1 -- 172.21.15.173:0/3574714901 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f440c009d00 con 0x7f44141021b0 2024-03-26T02:00:33.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.823+0000 7f441259c700 1 --2- 172.21.15.173:0/3574714901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4414100fb0 0x7f441406b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:00:33.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.824+0000 7f4411d9b700 1 --2- 172.21.15.173:0/3574714901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44141021b0 0x7f441419c4f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f440800c9b0 tx=0x7f440800ccc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:00:33.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.824+0000 7f44037fe700 1 -- 172.21.15.173:0/3574714901 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4408007b00 con 0x7f44141021b0 2024-03-26T02:00:33.826 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.824+0000 7f44037fe700 1 -- 172.21.15.173:0/3574714901 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f440800ce50 con 0x7f44141021b0 2024-03-26T02:00:33.827 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.824+0000 7f4418bc1700 1 -- 172.21.15.173:0/3574714901 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f441419f850 con 0x7f44141021b0 2024-03-26T02:00:33.827 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.824+0000 7f44037fe700 1 -- 172.21.15.173:0/3574714901 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4408018660 con 0x7f44141021b0 2024-03-26T02:00:33.827 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.824+0000 7f4418bc1700 1 -- 172.21.15.173:0/3574714901 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f441419fd50 con 0x7f44141021b0 2024-03-26T02:00:33.827 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.825+0000 7f4418bc1700 1 -- 172.21.15.173:0/3574714901 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4414107b90 con 0x7f44141021b0 2024-03-26T02:00:33.828 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.825+0000 7f44037fe700 1 -- 172.21.15.173:0/3574714901 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 15) v1 ==== 91363+0+0 (secure 0 0 0) 0x7f440800f450 con 0x7f44141021b0 2024-03-26T02:00:33.828 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.826+0000 7f44037fe700 1 --2- 172.21.15.173:0/3574714901 >> v2:172.21.15.72:6800/2923394687 conn(0x7f43fc06d990 0x7f43fc06fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:00:33.828 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.826+0000 7f44037fe700 1 -- 172.21.15.173:0/3574714901 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 3789+0+0 (secure 0 0 0) 0x7f440808c880 con 0x7f44141021b0 2024-03-26T02:00:33.828 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.826+0000 7f441259c700 1 --2- 172.21.15.173:0/3574714901 >> v2:172.21.15.72:6800/2923394687 conn(0x7f43fc06d990 0x7f43fc06fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:00:33.828 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.826+0000 7f441259c700 1 --2- 172.21.15.173:0/3574714901 >> v2:172.21.15.72:6800/2923394687 conn(0x7f43fc06d990 0x7f43fc06fe50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4414102010 tx=0x7f440400a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:00:33.831 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:33.830+0000 7f44037fe700 1 -- 172.21.15.173:0/3574714901 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4408056b20 con 0x7f44141021b0 2024-03-26T02:00:34.032 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:34.031+0000 7f4418bc1700 1 -- 172.21.15.173:0/3574714901 --> v2:172.21.15.72:6800/2923394687 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f44140624c0 con 0x7f43fc06d990 2024-03-26T02:00:34.032 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:33 smithi173 ceph-mon[33839]: pgmap v163: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-26T02:00:34.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:33 smithi072 ceph-mon[30177]: pgmap v163: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-26T02:00:34.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:33 smithi072 ceph-mon[41619]: pgmap v163: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-03-26T02:00:34.896 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:34 smithi173 ceph-mon[33839]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:00:34.896 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:34 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:00:34.896 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:34 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:00:34.896 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:34 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:35.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:34 smithi072 ceph-mon[30177]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:00:35.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:34 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:00:35.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:34 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:00:35.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:34 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:35.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:34 smithi072 ceph-mon[41619]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:00:35.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:34 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T02:00:35.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:34 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T02:00:35.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:34 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:35.913 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:35 smithi173 ceph-mon[33839]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T02:00:36.218 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:35 smithi072 ceph-mon[30177]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T02:00:36.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:35 smithi072 ceph-mon[41619]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T02:00:37.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:36 smithi173 ceph-mon[33839]: from='client.? 172.21.15.173:0/460599586' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7dabfd5-7bcf-4f9b-947d-b84777af1ea1"}]: dispatch 2024-03-26T02:00:37.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:36 smithi173 ceph-mon[33839]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7dabfd5-7bcf-4f9b-947d-b84777af1ea1"}]: dispatch 2024-03-26T02:00:37.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:36 smithi173 ceph-mon[33839]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7dabfd5-7bcf-4f9b-947d-b84777af1ea1"}]': finished 2024-03-26T02:00:37.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:36 smithi173 ceph-mon[33839]: osdmap e51: 8 total, 7 up, 8 in 2024-03-26T02:00:37.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:36 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:00:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:36 smithi072 ceph-mon[30177]: from='client.? 172.21.15.173:0/460599586' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7dabfd5-7bcf-4f9b-947d-b84777af1ea1"}]: dispatch 2024-03-26T02:00:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:36 smithi072 ceph-mon[30177]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7dabfd5-7bcf-4f9b-947d-b84777af1ea1"}]: dispatch 2024-03-26T02:00:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:36 smithi072 ceph-mon[30177]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7dabfd5-7bcf-4f9b-947d-b84777af1ea1"}]': finished 2024-03-26T02:00:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:36 smithi072 ceph-mon[30177]: osdmap e51: 8 total, 7 up, 8 in 2024-03-26T02:00:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:36 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:00:37.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:36 smithi072 ceph-mon[41619]: from='client.? 172.21.15.173:0/460599586' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7dabfd5-7bcf-4f9b-947d-b84777af1ea1"}]: dispatch 2024-03-26T02:00:37.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:36 smithi072 ceph-mon[41619]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7dabfd5-7bcf-4f9b-947d-b84777af1ea1"}]: dispatch 2024-03-26T02:00:37.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:36 smithi072 ceph-mon[41619]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7dabfd5-7bcf-4f9b-947d-b84777af1ea1"}]': finished 2024-03-26T02:00:37.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:36 smithi072 ceph-mon[41619]: osdmap e51: 8 total, 7 up, 8 in 2024-03-26T02:00:37.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:36 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:00:38.062 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:37 smithi173 ceph-mon[33839]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:38.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:37 smithi173 ceph-mon[33839]: from='client.? 172.21.15.173:0/2304900654' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:00:38.120 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:37 smithi072 ceph-mon[30177]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:38.120 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:37 smithi072 ceph-mon[30177]: from='client.? 172.21.15.173:0/2304900654' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:00:38.121 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:37 smithi072 ceph-mon[41619]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:38.121 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:37 smithi072 ceph-mon[41619]: from='client.? 172.21.15.173:0/2304900654' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T02:00:40.232 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:40 smithi173 ceph-mon[33839]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:40.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:40 smithi072 ceph-mon[30177]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:40.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:40 smithi072 ceph-mon[41619]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:42 smithi072 ceph-mon[30177]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:42.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:42 smithi072 ceph-mon[41619]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:42 smithi173 ceph-mon[33839]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:44.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:44 smithi072 ceph-mon[30177]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:44.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:44 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T02:00:44.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:44 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:44.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:44 smithi072 ceph-mon[41619]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:44.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:44 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T02:00:44.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:44 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:44.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:44 smithi173 ceph-mon[33839]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:44.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:44 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T02:00:44.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:44 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:45.305 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:45 smithi173 ceph-mon[33839]: Deploying daemon osd.7 on smithi173 2024-03-26T02:00:45.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:45 smithi072 ceph-mon[30177]: Deploying daemon osd.7 on smithi173 2024-03-26T02:00:45.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:45 smithi072 ceph-mon[41619]: Deploying daemon osd.7 on smithi173 2024-03-26T02:00:46.275 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:46 smithi173 ceph-mon[33839]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:46.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:46 smithi072 ceph-mon[30177]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:46.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:46 smithi072 ceph-mon[41619]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:48.312 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:48 smithi173 ceph-mon[33839]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:48.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:48 smithi072 ceph-mon[30177]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:48.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:48 smithi072 ceph-mon[41619]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:49.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:49 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:49.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:49 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:49.967 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:49 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:00:49.967 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:49 smithi173 ceph-mon[33839]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:50.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:49 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:50.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:49 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:50.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:49 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:00:50.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:49 smithi072 ceph-mon[30177]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:50.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:49 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:50.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:49 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:50.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:49 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:00:50.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:49 smithi072 ceph-mon[41619]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:52.296 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:52 smithi173 ceph-mon[33839]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:52.296 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:52 smithi173 ceph-mon[33839]: from='osd.7 v2:172.21.15.173:6812/2776185825' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:00:52.296 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:52 smithi173 ceph-mon[33839]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:00:52.442 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:52.441+0000 7f44037fe700 1 -- 172.21.15.173:0/3574714901 <== mgr.14152 v2:172.21.15.72:6800/2923394687 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f44140624c0 con 0x7f43fc06d990 2024-03-26T02:00:52.442 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 7 on host 'smithi173' 2024-03-26T02:00:52.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:52.444+0000 7f4418bc1700 1 -- 172.21.15.173:0/3574714901 >> v2:172.21.15.72:6800/2923394687 conn(0x7f43fc06d990 msgr2=0x7f43fc06fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:00:52.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:52.444+0000 7f4418bc1700 1 --2- 172.21.15.173:0/3574714901 >> v2:172.21.15.72:6800/2923394687 conn(0x7f43fc06d990 0x7f43fc06fe50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4414102010 tx=0x7f440400a3b0 comp rx=0 tx=0).stop 2024-03-26T02:00:52.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:52.444+0000 7f4418bc1700 1 -- 172.21.15.173:0/3574714901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44141021b0 msgr2=0x7f441419c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:00:52.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:52.444+0000 7f4418bc1700 1 --2- 172.21.15.173:0/3574714901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44141021b0 0x7f441419c4f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f440800c9b0 tx=0x7f440800ccc0 comp rx=0 tx=0).stop 2024-03-26T02:00:52.445 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:52.444+0000 7f4418bc1700 1 -- 172.21.15.173:0/3574714901 shutdown_connections 2024-03-26T02:00:52.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:52.444+0000 7f4418bc1700 1 --2- 172.21.15.173:0/3574714901 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4414102b50 0x7f441419eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:52.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:52.444+0000 7f4418bc1700 1 --2- 172.21.15.173:0/3574714901 >> v2:172.21.15.72:6800/2923394687 conn(0x7f43fc06d990 0x7f43fc06fe50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:52.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:52.444+0000 7f4418bc1700 1 --2- 172.21.15.173:0/3574714901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4414100fb0 0x7f441406b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:52.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:52.444+0000 7f4418bc1700 1 --2- 172.21.15.173:0/3574714901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f44141021b0 0x7f441419c4f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:52.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:52.445+0000 7f4418bc1700 1 -- 172.21.15.173:0/3574714901 >> 172.21.15.173:0/3574714901 conn(0x7f44140fc570 msgr2=0x7f44140fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:00:52.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:52.445+0000 7f4418bc1700 1 -- 172.21.15.173:0/3574714901 shutdown_connections 2024-03-26T02:00:52.446 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:00:52.445+0000 7f4418bc1700 1 -- 172.21.15.173:0/3574714901 wait complete. 2024-03-26T02:00:52.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:52 smithi072 ceph-mon[30177]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:52.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:52 smithi072 ceph-mon[30177]: from='osd.7 v2:172.21.15.173:6812/2776185825' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:00:52.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:52 smithi072 ceph-mon[30177]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:00:52.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:52 smithi072 ceph-mon[41619]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:52.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:52 smithi072 ceph-mon[41619]: from='osd.7 v2:172.21.15.173:6812/2776185825' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:00:52.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:52 smithi072 ceph-mon[41619]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T02:00:53.118 DEBUG:teuthology.orchestra.run.smithi173:osd.7> sudo journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.7.service 2024-03-26T02:00:53.121 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-26T02:00:53.122 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-26T02:00:53.374 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:00:53.395 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:53 smithi173 ceph-mon[33839]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T02:00:53.395 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:53 smithi173 ceph-mon[33839]: from='osd.7 v2:172.21.15.173:6812/2776185825' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-26T02:00:53.395 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:53 smithi173 ceph-mon[33839]: osdmap e52: 8 total, 7 up, 8 in 2024-03-26T02:00:53.395 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:53 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:00:53.395 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:53 smithi173 ceph-mon[33839]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-26T02:00:53.395 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:53 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:53.395 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:53 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:53.396 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:53 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:53.396 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:53 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:53.396 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:00:53 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:00:53.185+0000 7fcebc81f700 -1 osd.7 0 waiting for initial osdmap 2024-03-26T02:00:53.396 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:00:53 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:00:53.191+0000 7fceb8612700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T02:00:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:53 smithi072 ceph-mon[30177]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T02:00:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:53 smithi072 ceph-mon[30177]: from='osd.7 v2:172.21.15.173:6812/2776185825' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-26T02:00:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:53 smithi072 ceph-mon[30177]: osdmap e52: 8 total, 7 up, 8 in 2024-03-26T02:00:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:53 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:00:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:53 smithi072 ceph-mon[30177]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-26T02:00:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:53 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:53 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:53 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:53 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:53 smithi072 ceph-mon[41619]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T02:00:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:53 smithi072 ceph-mon[41619]: from='osd.7 v2:172.21.15.173:6812/2776185825' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-26T02:00:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:53 smithi072 ceph-mon[41619]: osdmap e52: 8 total, 7 up, 8 in 2024-03-26T02:00:53.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:53 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:00:53.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:53 smithi072 ceph-mon[41619]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-03-26T02:00:53.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:53 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:53.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:53 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:53.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:53 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:53.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:53 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:54.423 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:54 smithi173 ceph-mon[33839]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:54.423 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:54 smithi173 ceph-mon[33839]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-26T02:00:54.423 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:54 smithi173 ceph-mon[33839]: osdmap e53: 8 total, 7 up, 8 in 2024-03-26T02:00:54.423 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:54 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:00:54.423 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:54 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:00:54.423 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:54 smithi173 ceph-mon[33839]: osd.7 v2:172.21.15.173:6812/2776185825 boot 2024-03-26T02:00:54.423 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:54 smithi173 ceph-mon[33839]: osdmap e54: 8 total, 8 up, 8 in 2024-03-26T02:00:54.423 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:54 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:00:54.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:54 smithi072 ceph-mon[30177]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:54.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:54 smithi072 ceph-mon[30177]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-26T02:00:54.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:54 smithi072 ceph-mon[30177]: osdmap e53: 8 total, 7 up, 8 in 2024-03-26T02:00:54.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:54 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:00:54.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:54 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:00:54.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:54 smithi072 ceph-mon[30177]: osd.7 v2:172.21.15.173:6812/2776185825 boot 2024-03-26T02:00:54.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:54 smithi072 ceph-mon[30177]: osdmap e54: 8 total, 8 up, 8 in 2024-03-26T02:00:54.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:54 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:00:54.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:54 smithi072 ceph-mon[41619]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-03-26T02:00:54.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:54 smithi072 ceph-mon[41619]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-03-26T02:00:54.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:54 smithi072 ceph-mon[41619]: osdmap e53: 8 total, 7 up, 8 in 2024-03-26T02:00:54.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:54 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:00:54.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:54 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:00:54.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:54 smithi072 ceph-mon[41619]: osd.7 v2:172.21.15.173:6812/2776185825 boot 2024-03-26T02:00:54.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:54 smithi072 ceph-mon[41619]: osdmap e54: 8 total, 8 up, 8 in 2024-03-26T02:00:54.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:54 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:00:55.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.117+0000 7f8b17233700 1 -- 172.21.15.72:0/2322965071 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b101032d0 msgr2=0x7f8b10103730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:00:55.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.117+0000 7f8b17233700 1 --2- 172.21.15.72:0/2322965071 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b101032d0 0x7f8b10103730 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8afc009920 tx=0x7f8afc009c30 comp rx=0 tx=0).stop 2024-03-26T02:00:55.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.117+0000 7f8b17233700 1 -- 172.21.15.72:0/2322965071 shutdown_connections 2024-03-26T02:00:55.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.117+0000 7f8b17233700 1 --2- 172.21.15.72:0/2322965071 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8b10103c70 0x7f8b1010a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:55.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.117+0000 7f8b17233700 1 --2- 172.21.15.72:0/2322965071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b101020d0 0x7f8b101024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:55.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.117+0000 7f8b17233700 1 --2- 172.21.15.72:0/2322965071 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b101032d0 0x7f8b10103730 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:55.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.117+0000 7f8b17233700 1 -- 172.21.15.72:0/2322965071 >> 172.21.15.72:0/2322965071 conn(0x7f8b100fd690 msgr2=0x7f8b100ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:00:55.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.117+0000 7f8b17233700 1 -- 172.21.15.72:0/2322965071 shutdown_connections 2024-03-26T02:00:55.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.117+0000 7f8b17233700 1 -- 172.21.15.72:0/2322965071 wait complete. 2024-03-26T02:00:55.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.118+0000 7f8b17233700 1 Processor -- start 2024-03-26T02:00:55.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.118+0000 7f8b17233700 1 -- start start 2024-03-26T02:00:55.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.118+0000 7f8b17233700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b101020d0 0x7f8b10075e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:00:55.120 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.118+0000 7f8b17233700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8b101032d0 0x7f8b10074470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:00:55.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.118+0000 7f8b17233700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b10103c70 0x7f8b100749b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:00:55.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.118+0000 7f8b17233700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b10075130 con 0x7f8b10103c70 2024-03-26T02:00:55.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.118+0000 7f8b17233700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b100752a0 con 0x7f8b101020d0 2024-03-26T02:00:55.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.118+0000 7f8b17233700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b101a1840 con 0x7f8b101032d0 2024-03-26T02:00:55.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.118+0000 7f8b0ffff700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8b101032d0 0x7f8b10074470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:00:55.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.118+0000 7f8b0ffff700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8b101032d0 0x7f8b10074470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:53018/0 (socket says 172.21.15.72:53018) 2024-03-26T02:00:55.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.118+0000 7f8b0ffff700 1 -- 172.21.15.72:0/3551824718 learned_addr learned my addr 172.21.15.72:0/3551824718 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:00:55.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.118+0000 7f8b157d0700 1 --2- 172.21.15.72:0/3551824718 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b10103c70 0x7f8b100749b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:00:55.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.118+0000 7f8b14fcf700 1 --2- 172.21.15.72:0/3551824718 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b101020d0 0x7f8b10075e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:00:55.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.118+0000 7f8b0ffff700 1 -- 172.21.15.72:0/3551824718 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b101020d0 msgr2=0x7f8b10075e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:00:55.121 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.118+0000 7f8b0ffff700 1 --2- 172.21.15.72:0/3551824718 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b101020d0 0x7f8b10075e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:55.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.118+0000 7f8b0ffff700 1 -- 172.21.15.72:0/3551824718 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b10103c70 msgr2=0x7f8b100749b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:00:55.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.118+0000 7f8b0ffff700 1 --2- 172.21.15.72:0/3551824718 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b10103c70 0x7f8b100749b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:55.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.118+0000 7f8b0ffff700 1 -- 172.21.15.72:0/3551824718 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b08009d00 con 0x7f8b101032d0 2024-03-26T02:00:55.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.118+0000 7f8b14fcf700 1 --2- 172.21.15.72:0/3551824718 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b101020d0 0x7f8b10075e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T02:00:55.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.119+0000 7f8b0ffff700 1 --2- 172.21.15.72:0/3551824718 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8b101032d0 0x7f8b10074470 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8afc009840 tx=0x7f8afc00b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:00:55.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.119+0000 7f8b0dffb700 1 -- 172.21.15.72:0/3551824718 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc01c070 con 0x7f8b101032d0 2024-03-26T02:00:55.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.119+0000 7f8b0dffb700 1 -- 172.21.15.72:0/3551824718 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f8afc00f7f0 con 0x7f8b101032d0 2024-03-26T02:00:55.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.119+0000 7f8b17233700 1 -- 172.21.15.72:0/3551824718 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8afc009520 con 0x7f8b101032d0 2024-03-26T02:00:55.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.119+0000 7f8b0dffb700 1 -- 172.21.15.72:0/3551824718 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8afc003a70 con 0x7f8b101032d0 2024-03-26T02:00:55.122 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.119+0000 7f8b17233700 1 -- 172.21.15.72:0/3551824718 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b101a1da0 con 0x7f8b101032d0 2024-03-26T02:00:55.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.120+0000 7f8b17233700 1 -- 172.21.15.72:0/3551824718 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b10109490 con 0x7f8b101032d0 2024-03-26T02:00:55.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.120+0000 7f8b0dffb700 1 -- 172.21.15.72:0/3551824718 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 15) v1 ==== 91363+0+0 (secure 0 0 0) 0x7f8afc020030 con 0x7f8b101032d0 2024-03-26T02:00:55.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.121+0000 7f8b0dffb700 1 --2- 172.21.15.72:0/3551824718 >> v2:172.21.15.72:6800/2923394687 conn(0x7f8b0006d990 0x7f8b0006fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:00:55.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.121+0000 7f8b0dffb700 1 -- 172.21.15.72:0/3551824718 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 4110+0+0 (secure 0 0 0) 0x7f8afc08dab0 con 0x7f8b101032d0 2024-03-26T02:00:55.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.121+0000 7f8b14fcf700 1 --2- 172.21.15.72:0/3551824718 >> v2:172.21.15.72:6800/2923394687 conn(0x7f8b0006d990 0x7f8b0006fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:00:55.123 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.121+0000 7f8b14fcf700 1 --2- 172.21.15.72:0/3551824718 >> v2:172.21.15.72:6800/2923394687 conn(0x7f8b0006d990 0x7f8b0006fe50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8b04005950 tx=0x7f8b040058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:00:55.126 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.126+0000 7f8b0dffb700 1 -- 172.21.15.72:0/3551824718 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f8afc057c90 con 0x7f8b101032d0 2024-03-26T02:00:55.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.322+0000 7f8b17233700 1 -- 172.21.15.72:0/3551824718 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f8b1010d1c0 con 0x7f8b101032d0 2024-03-26T02:00:55.323 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.322+0000 7f8b0dffb700 1 -- 172.21.15.72:0/3551824718 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v55) v1 ==== 74+0+130 (secure 0 0 0) 0x7f8afc05b350 con 0x7f8b101032d0 2024-03-26T02:00:55.323 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T02:00:55.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.324+0000 7f8b17233700 1 -- 172.21.15.72:0/3551824718 >> v2:172.21.15.72:6800/2923394687 conn(0x7f8b0006d990 msgr2=0x7f8b0006fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:00:55.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.324+0000 7f8b17233700 1 --2- 172.21.15.72:0/3551824718 >> v2:172.21.15.72:6800/2923394687 conn(0x7f8b0006d990 0x7f8b0006fe50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8b04005950 tx=0x7f8b040058e0 comp rx=0 tx=0).stop 2024-03-26T02:00:55.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.324+0000 7f8b17233700 1 -- 172.21.15.72:0/3551824718 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8b101032d0 msgr2=0x7f8b10074470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:00:55.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.324+0000 7f8b17233700 1 --2- 172.21.15.72:0/3551824718 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8b101032d0 0x7f8b10074470 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8afc009840 tx=0x7f8afc00b790 comp rx=0 tx=0).stop 2024-03-26T02:00:55.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.325+0000 7f8b17233700 1 -- 172.21.15.72:0/3551824718 shutdown_connections 2024-03-26T02:00:55.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.325+0000 7f8b17233700 1 --2- 172.21.15.72:0/3551824718 >> v2:172.21.15.72:6800/2923394687 conn(0x7f8b0006d990 0x7f8b0006fe50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:55.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.325+0000 7f8b17233700 1 --2- 172.21.15.72:0/3551824718 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8b10103c70 0x7f8b100749b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:55.325 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.325+0000 7f8b17233700 1 --2- 172.21.15.72:0/3551824718 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8b101020d0 0x7f8b10075e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:55.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.325+0000 7f8b17233700 1 --2- 172.21.15.72:0/3551824718 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8b101032d0 0x7f8b10074470 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:55.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.325+0000 7f8b17233700 1 -- 172.21.15.72:0/3551824718 >> 172.21.15.72:0/3551824718 conn(0x7f8b100fd690 msgr2=0x7f8b101065d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:00:55.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.325+0000 7f8b17233700 1 -- 172.21.15.72:0/3551824718 shutdown_connections 2024-03-26T02:00:55.326 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:55.325+0000 7f8b17233700 1 -- 172.21.15.72:0/3551824718 wait complete. 2024-03-26T02:00:55.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:55 smithi072 ceph-mon[30177]: purged_snaps scrub starts 2024-03-26T02:00:55.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:55 smithi072 ceph-mon[30177]: purged_snaps scrub ok 2024-03-26T02:00:55.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:55 smithi072 ceph-mon[30177]: osdmap e55: 8 total, 8 up, 8 in 2024-03-26T02:00:55.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:55 smithi072 ceph-mon[30177]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:00:55.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:55 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3551824718' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T02:00:55.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:55 smithi072 ceph-mon[41619]: purged_snaps scrub starts 2024-03-26T02:00:55.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:55 smithi072 ceph-mon[41619]: purged_snaps scrub ok 2024-03-26T02:00:55.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:55 smithi072 ceph-mon[41619]: osdmap e55: 8 total, 8 up, 8 in 2024-03-26T02:00:55.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:55 smithi072 ceph-mon[41619]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:00:55.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:55 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3551824718' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T02:00:56.056 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1711418453,"num_in_osds":8,"osd_in_since":1711418436,"num_remapped_pgs":1} 2024-03-26T02:00:56.056 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T02:00:56.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:55 smithi173 ceph-mon[33839]: purged_snaps scrub starts 2024-03-26T02:00:56.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:55 smithi173 ceph-mon[33839]: purged_snaps scrub ok 2024-03-26T02:00:56.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:55 smithi173 ceph-mon[33839]: osdmap e55: 8 total, 8 up, 8 in 2024-03-26T02:00:56.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:55 smithi173 ceph-mon[33839]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:00:56.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:55 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3551824718' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T02:00:56.302 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:00:56.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[30177]: osdmap e56: 8 total, 8 up, 8 in 2024-03-26T02:00:56.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[30177]: Detected new or changed devices on smithi173 2024-03-26T02:00:56.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:56.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:56.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:56.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:56.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:56.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:56.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[30177]: Adjusting osd_memory_target on smithi173 to 4272M 2024-03-26T02:00:56.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:56.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:56.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:00:56.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:56.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:00:56.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:56.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:00:56.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:56.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[41619]: osdmap e56: 8 total, 8 up, 8 in 2024-03-26T02:00:56.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[41619]: Detected new or changed devices on smithi173 2024-03-26T02:00:56.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:56.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:56.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:56.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:56.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:56.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:56.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[41619]: Adjusting osd_memory_target on smithi173 to 4272M 2024-03-26T02:00:56.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:56.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:56.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:00:56.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:56.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:00:56.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:56.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:00:56.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:56 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:57.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:56 smithi173 ceph-mon[33839]: osdmap e56: 8 total, 8 up, 8 in 2024-03-26T02:00:57.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:56 smithi173 ceph-mon[33839]: Detected new or changed devices on smithi173 2024-03-26T02:00:57.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:56 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:57.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:56 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:57.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:56 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:57.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:56 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:57.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:56 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:57.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:56 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:00:57.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:56 smithi173 ceph-mon[33839]: Adjusting osd_memory_target on smithi173 to 4272M 2024-03-26T02:00:57.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:56 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:57.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:56 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:57.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:56 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:00:57.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:56 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:57.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:56 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:00:57.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:56 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:00:57.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:56 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:00:57.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:56 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:00:57.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.861+0000 7f9b49ffc700 1 -- 172.21.15.72:0/2517720658 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b44101b80 msgr2=0x7f9b44105bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:00:57.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.861+0000 7f9b49ffc700 1 --2- 172.21.15.72:0/2517720658 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b44101b80 0x7f9b44105bb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9b38009990 tx=0x7f9b38009ca0 comp rx=0 tx=0).stop 2024-03-26T02:00:57.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.861+0000 7f9b49ffc700 1 -- 172.21.15.72:0/2517720658 shutdown_connections 2024-03-26T02:00:57.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.861+0000 7f9b49ffc700 1 --2- 172.21.15.72:0/2517720658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b44106290 0x7f9b4410de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:57.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.861+0000 7f9b49ffc700 1 --2- 172.21.15.72:0/2517720658 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9b441011d0 0x7f9b441015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:57.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.861+0000 7f9b49ffc700 1 --2- 172.21.15.72:0/2517720658 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b44101b80 0x7f9b44105bb0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:57.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.861+0000 7f9b49ffc700 1 -- 172.21.15.72:0/2517720658 >> 172.21.15.72:0/2517720658 conn(0x7f9b44077ef0 msgr2=0x7f9b44078300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:00:57.864 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.861+0000 7f9b49ffc700 1 -- 172.21.15.72:0/2517720658 shutdown_connections 2024-03-26T02:00:57.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.861+0000 7f9b49ffc700 1 -- 172.21.15.72:0/2517720658 wait complete. 2024-03-26T02:00:57.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.862+0000 7f9b49ffc700 1 Processor -- start 2024-03-26T02:00:57.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.862+0000 7f9b49ffc700 1 -- start start 2024-03-26T02:00:57.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.862+0000 7f9b49ffc700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9b441011d0 0x7f9b4419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:00:57.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.862+0000 7f9b49ffc700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b44101b80 0x7f9b4419ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:00:57.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.862+0000 7f9b49ffc700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b44106290 0x7f9b44196ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:00:57.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.862+0000 7f9b49ffc700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b44197140 con 0x7f9b44106290 2024-03-26T02:00:57.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.862+0000 7f9b49ffc700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b44197280 con 0x7f9b44101b80 2024-03-26T02:00:57.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.862+0000 7f9b49ffc700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b44197440 con 0x7f9b441011d0 2024-03-26T02:00:57.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.862+0000 7f9b437fe700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9b441011d0 0x7f9b4419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:00:57.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.863+0000 7f9b43fff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b44106290 0x7f9b44196ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:00:57.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.863+0000 7f9b437fe700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9b441011d0 0x7f9b4419c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:53046/0 (socket says 172.21.15.72:53046) 2024-03-26T02:00:57.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.863+0000 7f9b437fe700 1 -- 172.21.15.72:0/3020879320 learned_addr learned my addr 172.21.15.72:0/3020879320 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:00:57.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.863+0000 7f9b42ffd700 1 --2- 172.21.15.72:0/3020879320 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b44101b80 0x7f9b4419ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:00:57.865 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.863+0000 7f9b43fff700 1 -- 172.21.15.72:0/3020879320 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9b441011d0 msgr2=0x7f9b4419c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:00:57.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.863+0000 7f9b43fff700 1 --2- 172.21.15.72:0/3020879320 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9b441011d0 0x7f9b4419c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:57.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.863+0000 7f9b43fff700 1 -- 172.21.15.72:0/3020879320 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b44101b80 msgr2=0x7f9b4419ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:00:57.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.863+0000 7f9b43fff700 1 --2- 172.21.15.72:0/3020879320 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b44101b80 0x7f9b4419ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:57.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.863+0000 7f9b43fff700 1 -- 172.21.15.72:0/3020879320 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b34009520 con 0x7f9b44106290 2024-03-26T02:00:57.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.863+0000 7f9b437fe700 1 --2- 172.21.15.72:0/3020879320 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9b441011d0 0x7f9b4419c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:00:57.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.863+0000 7f9b42ffd700 1 --2- 172.21.15.72:0/3020879320 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b44101b80 0x7f9b4419ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T02:00:57.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.863+0000 7f9b43fff700 1 --2- 172.21.15.72:0/3020879320 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b44106290 0x7f9b44196ae0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009ed0 tx=0x7f9b3c009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:00:57.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.863+0000 7f9b40ff9700 1 -- 172.21.15.72:0/3020879320 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c00ee40 con 0x7f9b44106290 2024-03-26T02:00:57.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.863+0000 7f9b40ff9700 1 -- 172.21.15.72:0/3020879320 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f9b3c0168f0 con 0x7f9b44106290 2024-03-26T02:00:57.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.863+0000 7f9b49ffc700 1 -- 172.21.15.72:0/3020879320 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b380095f0 con 0x7f9b44106290 2024-03-26T02:00:57.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.863+0000 7f9b40ff9700 1 -- 172.21.15.72:0/3020879320 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3c014970 con 0x7f9b44106290 2024-03-26T02:00:57.866 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.863+0000 7f9b49ffc700 1 -- 172.21.15.72:0/3020879320 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b4406a720 con 0x7f9b44106290 2024-03-26T02:00:57.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.864+0000 7f9b49ffc700 1 -- 172.21.15.72:0/3020879320 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b44102690 con 0x7f9b44106290 2024-03-26T02:00:57.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.865+0000 7f9b40ff9700 1 -- 172.21.15.72:0/3020879320 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 15) v1 ==== 91363+0+0 (secure 0 0 0) 0x7f9b3c016440 con 0x7f9b44106290 2024-03-26T02:00:57.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.865+0000 7f9b40ff9700 1 --2- 172.21.15.72:0/3020879320 >> v2:172.21.15.72:6800/2923394687 conn(0x7f9b2c06d990 0x7f9b2c06fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:00:57.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.865+0000 7f9b40ff9700 1 -- 172.21.15.72:0/3020879320 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4081+0+0 (secure 0 0 0) 0x7f9b3c006e40 con 0x7f9b44106290 2024-03-26T02:00:57.867 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.865+0000 7f9b437fe700 1 --2- 172.21.15.72:0/3020879320 >> v2:172.21.15.72:6800/2923394687 conn(0x7f9b2c06d990 0x7f9b2c06fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:00:57.868 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.865+0000 7f9b437fe700 1 --2- 172.21.15.72:0/3020879320 >> v2:172.21.15.72:6800/2923394687 conn(0x7f9b2c06d990 0x7f9b2c06fe50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9b44107630 tx=0x7f9b34009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:00:57.870 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:57.870+0000 7f9b40ff9700 1 -- 172.21.15.72:0/3020879320 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f9b3c058720 con 0x7f9b44106290 2024-03-26T02:00:57.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:57 smithi072 ceph-mon[30177]: osdmap e57: 8 total, 8 up, 8 in 2024-03-26T02:00:57.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:57 smithi072 ceph-mon[30177]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:00:57.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:57 smithi072 ceph-mon[41619]: osdmap e57: 8 total, 8 up, 8 in 2024-03-26T02:00:57.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:57 smithi072 ceph-mon[41619]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:00:58.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:57 smithi173 ceph-mon[33839]: osdmap e57: 8 total, 8 up, 8 in 2024-03-26T02:00:58.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:57 smithi173 ceph-mon[33839]: pgmap v182: 1 pgs: 1 remapped+peering; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:00:58.065 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:58.064+0000 7f9b49ffc700 1 -- 172.21.15.72:0/3020879320 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b44197fc0 con 0x7f9b44106290 2024-03-26T02:00:58.066 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:58.065+0000 7f9b40ff9700 1 -- 172.21.15.72:0/3020879320 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v58) v1 ==== 74+0+11503 (secure 0 0 0) 0x7f9b3c05bde0 con 0x7f9b44106290 2024-03-26T02:00:58.066 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T02:00:58.066 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":58,"fsid":"c86fbdd4-eb13-11ee-95d0-87774f69a715","created":"2024-03-26T01:55:31.587056+0000","modified":"2024-03-26T02:00:57.775675+0000","last_up_change":"2024-03-26T02:00:53.755156+0000","last_in_change":"2024-03-26T02:00:36.756317+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T01:58:47.401060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f311af9-e18d-42fa-9032-8fa8081d2c4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6801","nonce":1438405769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1438405769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1438405769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6803","nonce":1438405769}]},"public_addr":"172.21.15.72:6801/1438405769","cluster_addr":"172.21.15.72:6802/1438405769","heartbeat_back_addr":"172.21.15.72:6804/1438405769","heartbeat_front_addr":"172.21.15.72:6803/1438405769","state":["exists","up"]},{"osd":1,"uuid":"9db9d82b-f680-418a-9192-88eb5fe854b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6805","nonce":2073231730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2073231730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":2073231730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6807","nonce":2073231730}]},"public_addr":"172.21.15.72:6805/2073231730","cluster_addr":"172.21.15.72:6806/2073231730","heartbeat_back_addr":"172.21.15.72:6808/2073231730","heartbeat_front_addr":"172.21.15.72:6807/2073231730","state":["exists","up"]},{"osd":2,"uuid":"65c5c06e-e84d-4b5a-8204-65399b224f1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6809","nonce":3459293761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3459293761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3459293761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6811","nonce":3459293761}]},"public_addr":"172.21.15.72:6809/3459293761","cluster_addr":"172.21.15.72:6810/3459293761","heartbeat_back_addr":"172.21.15.72:6812/3459293761","heartbeat_front_addr":"172.21.15.72:6811/3459293761","state":["exists","up"]},{"osd":3,"uuid":"4f2f428c-7a84-4a34-8f42-179cbf06778d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6813","nonce":2677886334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":2677886334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":2677886334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6815","nonce":2677886334}]},"public_addr":"172.21.15.72:6813/2677886334","cluster_addr":"172.21.15.72:6814/2677886334","heartbeat_back_addr":"172.21.15.72:6816/2677886334","heartbeat_front_addr":"172.21.15.72:6815/2677886334","state":["exists","up"]},{"osd":4,"uuid":"5b3e8749-94e2-4727-9d1e-075ef3e43efd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":510505310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6801","nonce":510505310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6803","nonce":510505310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":510505310}]},"public_addr":"172.21.15.173:6800/510505310","cluster_addr":"172.21.15.173:6801/510505310","heartbeat_back_addr":"172.21.15.173:6803/510505310","heartbeat_front_addr":"172.21.15.173:6802/510505310","state":["exists","up"]},{"osd":5,"uuid":"652ab183-9759-426c-83cd-c61a676b06c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1295375838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6805","nonce":1295375838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6807","nonce":1295375838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1295375838}]},"public_addr":"172.21.15.173:6804/1295375838","cluster_addr":"172.21.15.173:6805/1295375838","heartbeat_back_addr":"172.21.15.173:6807/1295375838","heartbeat_front_addr":"172.21.15.173:6806/1295375838","state":["exists","up"]},{"osd":6,"uuid":"7d77483d-4aec-4685-8048-ca58d3482de3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3732303478}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6809","nonce":3732303478}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6811","nonce":3732303478}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3732303478}]},"public_addr":"172.21.15.173:6808/3732303478","cluster_addr":"172.21.15.173:6809/3732303478","heartbeat_back_addr":"172.21.15.173:6811/3732303478","heartbeat_front_addr":"172.21.15.173:6810/3732303478","state":["exists","up"]},{"osd":7,"uuid":"a7dabfd5-7bcf-4f9b-947d-b84777af1ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2776185825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6813","nonce":2776185825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6815","nonce":2776185825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2776185825}]},"public_addr":"172.21.15.173:6812/2776185825","cluster_addr":"172.21.15.173:6813/2776185825","heartbeat_back_addr":"172.21.15.173:6815/2776185825","heartbeat_front_addr":"172.21.15.173:6814/2776185825","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:57:52.811815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:58:19.048706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:58:44.908816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:59:10.323471+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:59:34.072843+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:00:00.403810+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:00:26.925604+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:00:53.049477+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/2939358967":"2024-03-27T01:56:27.069870+0000","172.21.15.72:0/334665634":"2024-03-27T01:56:27.069870+0000","172.21.15.72:0/3270791191":"2024-03-27T01:56:00.482837+0000","172.21.15.72:0/116986624":"2024-03-27T01:56:27.069870+0000","172.21.15.72:6800/347688983":"2024-03-27T01:56:00.482837+0000","172.21.15.72:0/2616932429":"2024-03-27T01:56:00.482837+0000","172.21.15.72:6800/3912119602":"2024-03-27T01:56:27.069870+0000","172.21.15.72:0/3486073291":"2024-03-27T01:56:00.482837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T02:00:58.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:58.067+0000 7f9b49ffc700 1 -- 172.21.15.72:0/3020879320 >> v2:172.21.15.72:6800/2923394687 conn(0x7f9b2c06d990 msgr2=0x7f9b2c06fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:00:58.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:58.067+0000 7f9b49ffc700 1 --2- 172.21.15.72:0/3020879320 >> v2:172.21.15.72:6800/2923394687 conn(0x7f9b2c06d990 0x7f9b2c06fe50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9b44107630 tx=0x7f9b34009260 comp rx=0 tx=0).stop 2024-03-26T02:00:58.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:58.067+0000 7f9b49ffc700 1 -- 172.21.15.72:0/3020879320 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b44106290 msgr2=0x7f9b44196ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:00:58.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:58.067+0000 7f9b49ffc700 1 --2- 172.21.15.72:0/3020879320 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b44106290 0x7f9b44196ae0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c009ed0 tx=0x7f9b3c009f00 comp rx=0 tx=0).stop 2024-03-26T02:00:58.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:58.067+0000 7f9b49ffc700 1 -- 172.21.15.72:0/3020879320 shutdown_connections 2024-03-26T02:00:58.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:58.067+0000 7f9b49ffc700 1 --2- 172.21.15.72:0/3020879320 >> v2:172.21.15.72:6800/2923394687 conn(0x7f9b2c06d990 0x7f9b2c06fe50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:58.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:58.067+0000 7f9b49ffc700 1 --2- 172.21.15.72:0/3020879320 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9b44106290 0x7f9b44196ae0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:58.068 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:58.067+0000 7f9b49ffc700 1 --2- 172.21.15.72:0/3020879320 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9b441011d0 0x7f9b4419c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:58.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:58.067+0000 7f9b49ffc700 1 --2- 172.21.15.72:0/3020879320 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9b44101b80 0x7f9b4419ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:00:58.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:58.067+0000 7f9b49ffc700 1 -- 172.21.15.72:0/3020879320 >> 172.21.15.72:0/3020879320 conn(0x7f9b44077ef0 msgr2=0x7f9b440fede0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:00:58.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:58.068+0000 7f9b49ffc700 1 -- 172.21.15.72:0/3020879320 shutdown_connections 2024-03-26T02:00:58.069 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:00:58.068+0000 7f9b49ffc700 1 -- 172.21.15.72:0/3020879320 wait complete. 2024-03-26T02:00:58.664 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T01:58:47.401060+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T02:00:58.664 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-26T02:00:58.903 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:00:58.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:58 smithi072 ceph-mon[41619]: osdmap e58: 8 total, 8 up, 8 in 2024-03-26T02:00:58.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3020879320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:00:58.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:58 smithi072 ceph-mon[30177]: osdmap e58: 8 total, 8 up, 8 in 2024-03-26T02:00:58.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3020879320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:00:59.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:58 smithi173 ceph-mon[33839]: osdmap e58: 8 total, 8 up, 8 in 2024-03-26T02:00:59.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3020879320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:01:00.033 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:00:59 smithi072 ceph-mon[30177]: pgmap v184: 1 pgs: 1 remapped+peering; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:01:00.033 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:00:59 smithi072 ceph-mon[41619]: pgmap v184: 1 pgs: 1 remapped+peering; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:01:00.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:00:59 smithi173 ceph-mon[33839]: pgmap v184: 1 pgs: 1 remapped+peering; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:01:00.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.563+0000 7f07d334f700 1 -- 172.21.15.72:0/3069079423 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07cc101920 msgr2=0x7f07cc10de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:00.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.563+0000 7f07d334f700 1 --2- 172.21.15.72:0/3069079423 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07cc101920 0x7f07cc10de00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f07c400b0a0 tx=0x7f07c400b3b0 comp rx=0 tx=0).stop 2024-03-26T02:01:00.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.563+0000 7f07d334f700 1 -- 172.21.15.72:0/3069079423 shutdown_connections 2024-03-26T02:01:00.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.563+0000 7f07d334f700 1 --2- 172.21.15.72:0/3069079423 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07cc101920 0x7f07cc10de00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:00.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.563+0000 7f07d334f700 1 --2- 172.21.15.72:0/3069079423 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07cc106fc0 0x7f07cc1073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:00.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.563+0000 7f07d334f700 1 --2- 172.21.15.72:0/3069079423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07cc100f80 0x7f07cc1013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:00.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.563+0000 7f07d334f700 1 -- 172.21.15.72:0/3069079423 >> 172.21.15.72:0/3069079423 conn(0x7f07cc077eb0 msgr2=0x7f07cc0782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:00.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.563+0000 7f07d334f700 1 -- 172.21.15.72:0/3069079423 shutdown_connections 2024-03-26T02:01:00.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.563+0000 7f07d334f700 1 -- 172.21.15.72:0/3069079423 wait complete. 2024-03-26T02:01:00.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.564+0000 7f07d334f700 1 Processor -- start 2024-03-26T02:01:00.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.564+0000 7f07d334f700 1 -- start start 2024-03-26T02:01:00.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.564+0000 7f07d334f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07cc100f80 0x7f07cc199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:00.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.564+0000 7f07d334f700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07cc101920 0x7f07cc19a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:00.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.564+0000 7f07d334f700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07cc106fc0 0x7f07cc19e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:00.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.564+0000 7f07d334f700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07cc19ee20 con 0x7f07cc101920 2024-03-26T02:01:00.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.564+0000 7f07d334f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07cc19ef90 con 0x7f07cc100f80 2024-03-26T02:01:00.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.564+0000 7f07d334f700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07cc19f150 con 0x7f07cc106fc0 2024-03-26T02:01:00.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.564+0000 7f07d08ea700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07cc101920 0x7f07cc19a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:00.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.564+0000 7f07d18ec700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07cc106fc0 0x7f07cc19e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:00.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.564+0000 7f07d08ea700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07cc101920 0x7f07cc19a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:48934/0 (socket says 172.21.15.72:48934) 2024-03-26T02:01:00.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.564+0000 7f07d08ea700 1 -- 172.21.15.72:0/2032385240 learned_addr learned my addr 172.21.15.72:0/2032385240 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:01:00.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.564+0000 7f07d10eb700 1 --2- 172.21.15.72:0/2032385240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07cc100f80 0x7f07cc199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:00.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.564+0000 7f07d18ec700 1 -- 172.21.15.72:0/2032385240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07cc100f80 msgr2=0x7f07cc199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:00.567 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.564+0000 7f07d18ec700 1 --2- 172.21.15.72:0/2032385240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07cc100f80 0x7f07cc199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:00.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.564+0000 7f07d18ec700 1 -- 172.21.15.72:0/2032385240 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07cc101920 msgr2=0x7f07cc19a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:00.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.564+0000 7f07d18ec700 1 --2- 172.21.15.72:0/2032385240 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07cc101920 0x7f07cc19a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:00.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.564+0000 7f07d18ec700 1 -- 172.21.15.72:0/2032385240 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07bc009520 con 0x7f07cc106fc0 2024-03-26T02:01:00.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.564+0000 7f07d10eb700 1 --2- 172.21.15.72:0/2032385240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07cc100f80 0x7f07cc199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T02:01:00.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.564+0000 7f07d08ea700 1 --2- 172.21.15.72:0/2032385240 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07cc101920 0x7f07cc19a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:01:00.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.565+0000 7f07d18ec700 1 --2- 172.21.15.72:0/2032385240 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07cc106fc0 0x7f07cc19e810 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f07c400b0a0 tx=0x7f07c400be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:00.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.565+0000 7f07c27fc700 1 -- 172.21.15.72:0/2032385240 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c400e040 con 0x7f07cc106fc0 2024-03-26T02:01:00.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.565+0000 7f07c27fc700 1 -- 172.21.15.72:0/2032385240 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f07c401d470 con 0x7f07cc106fc0 2024-03-26T02:01:00.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.565+0000 7f07d334f700 1 -- 172.21.15.72:0/2032385240 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07c4009d00 con 0x7f07cc106fc0 2024-03-26T02:01:00.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.565+0000 7f07c27fc700 1 -- 172.21.15.72:0/2032385240 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c40052a0 con 0x7f07cc106fc0 2024-03-26T02:01:00.568 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.565+0000 7f07d334f700 1 -- 172.21.15.72:0/2032385240 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07cc1a6ac0 con 0x7f07cc106fc0 2024-03-26T02:01:00.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.566+0000 7f07d334f700 1 -- 172.21.15.72:0/2032385240 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07cc19f310 con 0x7f07cc106fc0 2024-03-26T02:01:00.569 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.566+0000 7f07c27fc700 1 -- 172.21.15.72:0/2032385240 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 15) v1 ==== 91363+0+0 (secure 0 0 0) 0x7f07c40074e0 con 0x7f07cc106fc0 2024-03-26T02:01:00.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.567+0000 7f07c27fc700 1 --2- 172.21.15.72:0/2032385240 >> v2:172.21.15.72:6800/2923394687 conn(0x7f07b806d990 0x7f07b806fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:00.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.567+0000 7f07c27fc700 1 -- 172.21.15.72:0/2032385240 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4081+0+0 (secure 0 0 0) 0x7f07c408dd00 con 0x7f07cc106fc0 2024-03-26T02:01:00.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.567+0000 7f07d10eb700 1 --2- 172.21.15.72:0/2032385240 >> v2:172.21.15.72:6800/2923394687 conn(0x7f07b806d990 0x7f07b806fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:00.570 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.567+0000 7f07d10eb700 1 --2- 172.21.15.72:0/2032385240 >> v2:172.21.15.72:6800/2923394687 conn(0x7f07b806d990 0x7f07b806fe50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f07bc009e20 tx=0x7f07bc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:00.572 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.571+0000 7f07c27fc700 1 -- 172.21.15.72:0/2032385240 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f07c4057ef0 con 0x7f07cc106fc0 2024-03-26T02:01:00.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.764+0000 7f07d334f700 1 -- 172.21.15.72:0/2032385240 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f07cc19b2a0 con 0x7f07cc106fc0 2024-03-26T02:01:00.764 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.764+0000 7f07c27fc700 1 -- 172.21.15.72:0/2032385240 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (secure 0 0 0) 0x7f07c405b5b0 con 0x7f07cc106fc0 2024-03-26T02:01:00.765 INFO:teuthology.orchestra.run.smithi072.stdout:pg_num: 1 2024-03-26T02:01:00.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.766+0000 7f07d334f700 1 -- 172.21.15.72:0/2032385240 >> v2:172.21.15.72:6800/2923394687 conn(0x7f07b806d990 msgr2=0x7f07b806fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:00.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.766+0000 7f07d334f700 1 --2- 172.21.15.72:0/2032385240 >> v2:172.21.15.72:6800/2923394687 conn(0x7f07b806d990 0x7f07b806fe50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f07bc009e20 tx=0x7f07bc009260 comp rx=0 tx=0).stop 2024-03-26T02:01:00.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.766+0000 7f07d334f700 1 -- 172.21.15.72:0/2032385240 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07cc106fc0 msgr2=0x7f07cc19e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:00.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.766+0000 7f07d334f700 1 --2- 172.21.15.72:0/2032385240 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07cc106fc0 0x7f07cc19e810 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f07c400b0a0 tx=0x7f07c400be80 comp rx=0 tx=0).stop 2024-03-26T02:01:00.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.766+0000 7f07d334f700 1 -- 172.21.15.72:0/2032385240 shutdown_connections 2024-03-26T02:01:00.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.766+0000 7f07d334f700 1 --2- 172.21.15.72:0/2032385240 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f07cc106fc0 0x7f07cc19e810 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:00.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.766+0000 7f07d334f700 1 --2- 172.21.15.72:0/2032385240 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f07cc100f80 0x7f07cc199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:00.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.766+0000 7f07d334f700 1 --2- 172.21.15.72:0/2032385240 >> v2:172.21.15.72:6800/2923394687 conn(0x7f07b806d990 0x7f07b806fe50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:00.767 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.766+0000 7f07d334f700 1 --2- 172.21.15.72:0/2032385240 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f07cc101920 0x7f07cc19a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:00.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.766+0000 7f07d334f700 1 -- 172.21.15.72:0/2032385240 >> 172.21.15.72:0/2032385240 conn(0x7f07cc077eb0 msgr2=0x7f07cc10c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:00.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.766+0000 7f07d334f700 1 -- 172.21.15.72:0/2032385240 shutdown_connections 2024-03-26T02:01:00.768 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:00.766+0000 7f07d334f700 1 -- 172.21.15.72:0/2032385240 wait complete. 2024-03-26T02:01:00.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2032385240' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T02:01:00.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2032385240' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T02:01:01.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2032385240' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T02:01:01.439 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi072 2024-03-26T02:01:01.439 DEBUG:teuthology.orchestra.run.smithi173:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi072=foo.a' 2024-03-26T02:01:01.688 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:01:02.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:01 smithi173 ceph-mon[33839]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T02:01:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:01 smithi072 ceph-mon[30177]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T02:01:02.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:01 smithi072 ceph-mon[41619]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T02:01:03.299 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.296+0000 7f536ddc6700 1 -- 172.21.15.173:0/1511162861 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5368100180 msgr2=0x7f5368100600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:03.299 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.296+0000 7f536ddc6700 1 --2- 172.21.15.173:0/1511162861 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5368100180 0x7f5368100600 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f535c009990 tx=0x7f535c009ca0 comp rx=0 tx=0).stop 2024-03-26T02:01:03.299 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.296+0000 7f536ddc6700 1 -- 172.21.15.173:0/1511162861 shutdown_connections 2024-03-26T02:01:03.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.296+0000 7f536ddc6700 1 --2- 172.21.15.173:0/1511162861 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5368100b40 0x7f536810a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:03.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.296+0000 7f536ddc6700 1 --2- 172.21.15.173:0/1511162861 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5368103700 0x7f5368103b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:03.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.296+0000 7f536ddc6700 1 --2- 172.21.15.173:0/1511162861 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5368100180 0x7f5368100600 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:03.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.296+0000 7f536ddc6700 1 -- 172.21.15.173:0/1511162861 >> 172.21.15.173:0/1511162861 conn(0x7f53680fbd20 msgr2=0x7f53680fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:03.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.296+0000 7f536ddc6700 1 -- 172.21.15.173:0/1511162861 shutdown_connections 2024-03-26T02:01:03.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.296+0000 7f536ddc6700 1 -- 172.21.15.173:0/1511162861 wait complete. 2024-03-26T02:01:03.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.297+0000 7f536ddc6700 1 Processor -- start 2024-03-26T02:01:03.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.297+0000 7f536ddc6700 1 -- start start 2024-03-26T02:01:03.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.297+0000 7f536ddc6700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5368100180 0x7f5368197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:03.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.297+0000 7f536ddc6700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5368100b40 0x7f5368197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:03.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.297+0000 7f536ddc6700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5368103700 0x7f536819ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:03.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.297+0000 7f536ddc6700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536819f2f0 con 0x7f5368100b40 2024-03-26T02:01:03.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.297+0000 7f536ddc6700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536819f430 con 0x7f5368100180 2024-03-26T02:01:03.300 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.297+0000 7f536ddc6700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f536819f5c0 con 0x7f5368103700 2024-03-26T02:01:03.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.297+0000 7f53677fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5368100180 0x7f5368197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:03.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.297+0000 7f53677fe700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5368100180 0x7f5368197760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:52456/0 (socket says 172.21.15.173:52456) 2024-03-26T02:01:03.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.297+0000 7f53677fe700 1 -- 172.21.15.173:0/2069879108 learned_addr learned my addr 172.21.15.173:0/2069879108 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-26T02:01:03.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.297+0000 7f5366ffd700 1 --2- 172.21.15.173:0/2069879108 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5368100b40 0x7f5368197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:03.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.297+0000 7f5367fff700 1 --2- 172.21.15.173:0/2069879108 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5368103700 0x7f536819ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:03.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.298+0000 7f53677fe700 1 -- 172.21.15.173:0/2069879108 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5368103700 msgr2=0x7f536819ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:03.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.298+0000 7f53677fe700 1 --2- 172.21.15.173:0/2069879108 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5368103700 0x7f536819ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:03.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.298+0000 7f53677fe700 1 -- 172.21.15.173:0/2069879108 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5368100b40 msgr2=0x7f5368197ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:03.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.298+0000 7f53677fe700 1 --2- 172.21.15.173:0/2069879108 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5368100b40 0x7f5368197ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:03.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.298+0000 7f53677fe700 1 -- 172.21.15.173:0/2069879108 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f535c0095f0 con 0x7f5368100180 2024-03-26T02:01:03.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.298+0000 7f53677fe700 1 --2- 172.21.15.173:0/2069879108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5368100180 0x7f5368197760 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f535800d980 tx=0x7f535800dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:03.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.298+0000 7f5364ff9700 1 -- 172.21.15.173:0/2069879108 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5358004020 con 0x7f5368100180 2024-03-26T02:01:03.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.298+0000 7f5364ff9700 1 -- 172.21.15.173:0/2069879108 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f535800f690 con 0x7f5368100180 2024-03-26T02:01:03.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.298+0000 7f536ddc6700 1 -- 172.21.15.173:0/2069879108 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f536819f8a0 con 0x7f5368100180 2024-03-26T02:01:03.301 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.298+0000 7f5364ff9700 1 -- 172.21.15.173:0/2069879108 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5358003a90 con 0x7f5368100180 2024-03-26T02:01:03.302 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.298+0000 7f536ddc6700 1 -- 172.21.15.173:0/2069879108 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f536819fdf0 con 0x7f5368100180 2024-03-26T02:01:03.302 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.299+0000 7f536ddc6700 1 -- 172.21.15.173:0/2069879108 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5368109590 con 0x7f5368100180 2024-03-26T02:01:03.302 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.299+0000 7f5364ff9700 1 -- 172.21.15.173:0/2069879108 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 15) v1 ==== 91363+0+0 (secure 0 0 0) 0x7f535800fb00 con 0x7f5368100180 2024-03-26T02:01:03.303 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.300+0000 7f5364ff9700 1 --2- 172.21.15.173:0/2069879108 >> v2:172.21.15.72:6800/2923394687 conn(0x7f535006d990 0x7f535006fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:03.303 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.300+0000 7f5364ff9700 1 -- 172.21.15.173:0/2069879108 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4081+0+0 (secure 0 0 0) 0x7f535808c4d0 con 0x7f5368100180 2024-03-26T02:01:03.303 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.300+0000 7f5366ffd700 1 --2- 172.21.15.173:0/2069879108 >> v2:172.21.15.72:6800/2923394687 conn(0x7f535006d990 0x7f535006fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:03.303 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.300+0000 7f5366ffd700 1 --2- 172.21.15.173:0/2069879108 >> v2:172.21.15.72:6800/2923394687 conn(0x7f535006d990 0x7f535006fe50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f535c00b5c0 tx=0x7f535c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:03.305 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.305+0000 7f5364ff9700 1 -- 172.21.15.173:0/2069879108 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f53580566c0 con 0x7f5368100180 2024-03-26T02:01:03.511 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.510+0000 7f536ddc6700 1 -- 172.21.15.173:0/2069879108 --> v2:172.21.15.72:6800/2923394687 -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi072=foo.a", "target": ["mon-mgr", ""]}) v1 -- 0x7f536810d4a0 con 0x7f535006d990 2024-03-26T02:01:03.518 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.518+0000 7f5364ff9700 1 -- 172.21.15.173:0/2069879108 <== mgr.14152 v2:172.21.15.72:6800/2923394687 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+30 (secure 0 0 0) 0x7f536810d4a0 con 0x7f535006d990 2024-03-26T02:01:03.519 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled rgw.foo.a update... 2024-03-26T02:01:03.521 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.520+0000 7f536ddc6700 1 -- 172.21.15.173:0/2069879108 >> v2:172.21.15.72:6800/2923394687 conn(0x7f535006d990 msgr2=0x7f535006fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:03.521 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.520+0000 7f536ddc6700 1 --2- 172.21.15.173:0/2069879108 >> v2:172.21.15.72:6800/2923394687 conn(0x7f535006d990 0x7f535006fe50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f535c00b5c0 tx=0x7f535c005fb0 comp rx=0 tx=0).stop 2024-03-26T02:01:03.521 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.520+0000 7f536ddc6700 1 -- 172.21.15.173:0/2069879108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5368100180 msgr2=0x7f5368197760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:03.521 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.520+0000 7f536ddc6700 1 --2- 172.21.15.173:0/2069879108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5368100180 0x7f5368197760 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f535800d980 tx=0x7f535800dc90 comp rx=0 tx=0).stop 2024-03-26T02:01:03.521 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.520+0000 7f536ddc6700 1 -- 172.21.15.173:0/2069879108 shutdown_connections 2024-03-26T02:01:03.521 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.520+0000 7f536ddc6700 1 --2- 172.21.15.173:0/2069879108 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5368103700 0x7f536819ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:03.522 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.520+0000 7f536ddc6700 1 --2- 172.21.15.173:0/2069879108 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5368100180 0x7f5368197760 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:03.522 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.520+0000 7f536ddc6700 1 --2- 172.21.15.173:0/2069879108 >> v2:172.21.15.72:6800/2923394687 conn(0x7f535006d990 0x7f535006fe50 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:03.522 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.520+0000 7f536ddc6700 1 --2- 172.21.15.173:0/2069879108 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5368100b40 0x7f5368197ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:03.522 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.520+0000 7f536ddc6700 1 -- 172.21.15.173:0/2069879108 >> 172.21.15.173:0/2069879108 conn(0x7f53680fbd20 msgr2=0x7f53681066d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:03.522 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.521+0000 7f536ddc6700 1 -- 172.21.15.173:0/2069879108 shutdown_connections 2024-03-26T02:01:03.522 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:03.521+0000 7f536ddc6700 1 -- 172.21.15.173:0/2069879108 wait complete. 2024-03-26T02:01:04.119 DEBUG:teuthology.orchestra.run.smithi072:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@rgw.foo.a.service 2024-03-26T02:01:04.122 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi173 2024-03-26T02:01:04.122 DEBUG:teuthology.orchestra.run.smithi173:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-26T02:01:04.360 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:01:04.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:04 smithi072 ceph-mon[41619]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-26T02:01:04.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:04 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:04.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:04 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:01:04.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:04 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:01:04.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:04 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:01:04.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:04 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:04.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:04 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:01:04.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:04 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T02:01:04.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:04 smithi072 ceph-mon[30177]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-26T02:01:04.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:04 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:04.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:04 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:01:04.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:04 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:01:04.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:04 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:01:04.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:04 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:04.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:04 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:01:04.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:04 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T02:01:04.481 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:04 smithi173 ceph-mon[33839]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-26T02:01:04.481 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:04 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:04.481 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:04 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:01:04.481 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:04 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:01:04.481 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:04 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:01:04.481 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:04 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:04.481 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:04 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T02:01:04.481 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:04 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T02:01:05.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:05 smithi072 ceph-mon[30177]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi072=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:05.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:05 smithi072 ceph-mon[30177]: Saving service rgw.foo.a spec with placement smithi072=foo.a;count:1 2024-03-26T02:01:05.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:05 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:05.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:05 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:01:05.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:05 smithi072 ceph-mon[30177]: Deploying daemon rgw.foo.a on smithi072 2024-03-26T02:01:05.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:05 smithi072 ceph-mon[41619]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi072=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:05.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:05 smithi072 ceph-mon[41619]: Saving service rgw.foo.a spec with placement smithi072=foo.a;count:1 2024-03-26T02:01:05.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:05 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:05.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:05 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:01:05.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:05 smithi072 ceph-mon[41619]: Deploying daemon rgw.foo.a on smithi072 2024-03-26T02:01:05.498 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:05 smithi173 ceph-mon[33839]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi072=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:05.498 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:05 smithi173 ceph-mon[33839]: Saving service rgw.foo.a spec with placement smithi072=foo.a;count:1 2024-03-26T02:01:05.499 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:05 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:05.499 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:05 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:01:05.499 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:05 smithi173 ceph-mon[33839]: Deploying daemon rgw.foo.a on smithi072 2024-03-26T02:01:05.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.991+0000 7fa52752d700 1 -- 172.21.15.173:0/1748879468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5201020b0 msgr2=0x7fa5201024d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:05.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.991+0000 7fa52752d700 1 --2- 172.21.15.173:0/1748879468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5201020b0 0x7fa5201024d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa51c0099c0 tx=0x7fa51c009cd0 comp rx=0 tx=0).stop 2024-03-26T02:01:05.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.991+0000 7fa52752d700 1 -- 172.21.15.173:0/1748879468 shutdown_connections 2024-03-26T02:01:05.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.991+0000 7fa52752d700 1 --2- 172.21.15.173:0/1748879468 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa520103c50 0x7fa52010a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:05.994 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.991+0000 7fa52752d700 1 --2- 172.21.15.173:0/1748879468 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa5201020b0 0x7fa5201024d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:05.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.991+0000 7fa52752d700 1 --2- 172.21.15.173:0/1748879468 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5201032b0 0x7fa520103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:05.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.991+0000 7fa52752d700 1 -- 172.21.15.173:0/1748879468 >> 172.21.15.173:0/1748879468 conn(0x7fa5200fd690 msgr2=0x7fa5200ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:05.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.991+0000 7fa52752d700 1 -- 172.21.15.173:0/1748879468 shutdown_connections 2024-03-26T02:01:05.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.991+0000 7fa52752d700 1 -- 172.21.15.173:0/1748879468 wait complete. 2024-03-26T02:01:05.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.992+0000 7fa52752d700 1 Processor -- start 2024-03-26T02:01:05.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.992+0000 7fa52752d700 1 -- start start 2024-03-26T02:01:05.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.992+0000 7fa52752d700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa5201020b0 0x7fa5201976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:05.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.992+0000 7fa52752d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5201032b0 0x7fa520197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:05.995 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.992+0000 7fa52752d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa520103c50 0x7fa52019eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:05.996 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.992+0000 7fa52752d700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52019f270 con 0x7fa5201032b0 2024-03-26T02:01:05.996 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.992+0000 7fa52752d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52019f3b0 con 0x7fa520103c50 2024-03-26T02:01:05.996 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.992+0000 7fa52752d700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa52019f540 con 0x7fa5201020b0 2024-03-26T02:01:05.996 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.992+0000 7fa525aca700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa520103c50 0x7fa52019eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:05.996 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.992+0000 7fa525aca700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa520103c50 0x7fa52019eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:52472/0 (socket says 172.21.15.173:52472) 2024-03-26T02:01:05.996 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.992+0000 7fa5252c9700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa5201020b0 0x7fa5201976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:05.996 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.992+0000 7fa525aca700 1 -- 172.21.15.173:0/3455689994 learned_addr learned my addr 172.21.15.173:0/3455689994 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-26T02:01:05.996 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.992+0000 7fa524ac8700 1 --2- 172.21.15.173:0/3455689994 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5201032b0 0x7fa520197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:05.997 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.992+0000 7fa525aca700 1 -- 172.21.15.173:0/3455689994 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa5201020b0 msgr2=0x7fa5201976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:05.997 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.992+0000 7fa525aca700 1 --2- 172.21.15.173:0/3455689994 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa5201020b0 0x7fa5201976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:05.997 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.992+0000 7fa525aca700 1 -- 172.21.15.173:0/3455689994 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5201032b0 msgr2=0x7fa520197c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:05.997 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.992+0000 7fa525aca700 1 --2- 172.21.15.173:0/3455689994 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5201032b0 0x7fa520197c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:05.997 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.992+0000 7fa525aca700 1 -- 172.21.15.173:0/3455689994 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51c0095f0 con 0x7fa520103c50 2024-03-26T02:01:05.997 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.993+0000 7fa525aca700 1 --2- 172.21.15.173:0/3455689994 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa520103c50 0x7fa52019eca0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa51800ada0 tx=0x7fa51800c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:05.997 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.993+0000 7fa5167fc700 1 -- 172.21.15.173:0/3455689994 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51800f070 con 0x7fa520103c50 2024-03-26T02:01:05.997 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.993+0000 7fa5167fc700 1 -- 172.21.15.173:0/3455689994 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fa518007b30 con 0x7fa520103c50 2024-03-26T02:01:05.998 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.993+0000 7fa5167fc700 1 -- 172.21.15.173:0/3455689994 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa518005280 con 0x7fa520103c50 2024-03-26T02:01:05.998 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.993+0000 7fa52752d700 1 -- 172.21.15.173:0/3455689994 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa52019f820 con 0x7fa520103c50 2024-03-26T02:01:05.998 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.993+0000 7fa52752d700 1 -- 172.21.15.173:0/3455689994 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa52019fd70 con 0x7fa520103c50 2024-03-26T02:01:05.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.994+0000 7fa52752d700 1 -- 172.21.15.173:0/3455689994 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa520109470 con 0x7fa520103c50 2024-03-26T02:01:05.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.994+0000 7fa5167fc700 1 -- 172.21.15.173:0/3455689994 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 15) v1 ==== 91363+0+0 (secure 0 0 0) 0x7fa51800c8d0 con 0x7fa520103c50 2024-03-26T02:01:05.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.995+0000 7fa5167fc700 1 --2- 172.21.15.173:0/3455689994 >> v2:172.21.15.72:6800/2923394687 conn(0x7fa50c06d990 0x7fa50c06fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:05.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.995+0000 7fa5167fc700 1 -- 172.21.15.173:0/3455689994 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 4081+0+0 (secure 0 0 0) 0x7fa51808bff0 con 0x7fa520103c50 2024-03-26T02:01:05.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.995+0000 7fa5252c9700 1 --2- 172.21.15.173:0/3455689994 >> v2:172.21.15.72:6800/2923394687 conn(0x7fa50c06d990 0x7fa50c06fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:05.999 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.995+0000 7fa5252c9700 1 --2- 172.21.15.173:0/3455689994 >> v2:172.21.15.72:6800/2923394687 conn(0x7fa50c06d990 0x7fa50c06fe50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa51c006010 tx=0x7fa51c005040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:06.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:05.999+0000 7fa5167fc700 1 -- 172.21.15.173:0/3455689994 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa5180561e0 con 0x7fa520103c50 2024-03-26T02:01:06.194 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:06.194+0000 7fa52752d700 1 -- 172.21.15.173:0/3455689994 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) v1 -- 0x7fa52010d160 con 0x7fa520103c50 2024-03-26T02:01:06.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:06 smithi072 ceph-mon[30177]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-26T02:01:06.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:06 smithi072 ceph-mon[41619]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-26T02:01:06.470 INFO:journalctl@ceph.rgw.foo.a.smithi072.stdout:Mar 26 02:01:06 smithi072 systemd[1]: Starting Ceph rgw.foo.a for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:01:06.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:06 smithi173 ceph-mon[33839]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-26T02:01:06.802 INFO:journalctl@ceph.rgw.foo.a.smithi072.stdout:Mar 26 02:01:06 smithi072 podman[70502]: 2024-03-26 02:01:06.413053918 +0000 UTC m=+0.017955438 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:01:06.803 INFO:journalctl@ceph.rgw.foo.a.smithi072.stdout:Mar 26 02:01:06 smithi072 podman[70502]: 2024-03-26 02:01:06.514980566 +0000 UTC m=+0.119882070 container create 6ef2c13736317266c695bfb57d41686b431bb699f2bb158de55fbf2986bb7a29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-rgw-foo-a, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-26T02:01:06.803 INFO:journalctl@ceph.rgw.foo.a.smithi072.stdout:Mar 26 02:01:06 smithi072 podman[70502]: 2024-03-26 02:01:06.735735687 +0000 UTC m=+0.340637191 container init 6ef2c13736317266c695bfb57d41686b431bb699f2bb158de55fbf2986bb7a29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-rgw-foo-a, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_CLEAN=True) 2024-03-26T02:01:06.803 INFO:journalctl@ceph.rgw.foo.a.smithi072.stdout:Mar 26 02:01:06 smithi072 podman[70502]: 2024-03-26 02:01:06.741253229 +0000 UTC m=+0.346154734 container start 6ef2c13736317266c695bfb57d41686b431bb699f2bb158de55fbf2986bb7a29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-rgw-foo-a, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-26T02:01:07.188 INFO:journalctl@ceph.rgw.foo.a.smithi072.stdout:Mar 26 02:01:06 smithi072 bash[70502]: 6ef2c13736317266c695bfb57d41686b431bb699f2bb158de55fbf2986bb7a29 2024-03-26T02:01:07.188 INFO:journalctl@ceph.rgw.foo.a.smithi072.stdout:Mar 26 02:01:06 smithi072 systemd[1]: Started Ceph rgw.foo.a for c86fbdd4-eb13-11ee-95d0-87774f69a715. 2024-03-26T02:01:07.195 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:07.194+0000 7fa5167fc700 1 -- 172.21.15.173:0/3455689994 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'datapool' created v59) v1 ==== 160+0+0 (secure 0 0 0) 0x7fa51800cb80 con 0x7fa520103c50 2024-03-26T02:01:07.199 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:07.199+0000 7fa52752d700 1 -- 172.21.15.173:0/3455689994 >> v2:172.21.15.72:6800/2923394687 conn(0x7fa50c06d990 msgr2=0x7fa50c06fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:07.199 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:07.199+0000 7fa52752d700 1 --2- 172.21.15.173:0/3455689994 >> v2:172.21.15.72:6800/2923394687 conn(0x7fa50c06d990 0x7fa50c06fe50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fa51c006010 tx=0x7fa51c005040 comp rx=0 tx=0).stop 2024-03-26T02:01:07.199 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:07.199+0000 7fa52752d700 1 -- 172.21.15.173:0/3455689994 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa520103c50 msgr2=0x7fa52019eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:07.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:07.199+0000 7fa52752d700 1 --2- 172.21.15.173:0/3455689994 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa520103c50 0x7fa52019eca0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa51800ada0 tx=0x7fa51800c330 comp rx=0 tx=0).stop 2024-03-26T02:01:07.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:07.199+0000 7fa52752d700 1 -- 172.21.15.173:0/3455689994 shutdown_connections 2024-03-26T02:01:07.200 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:07.199+0000 7fa52752d700 1 --2- 172.21.15.173:0/3455689994 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa520103c50 0x7fa52019eca0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:07.201 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:07.199+0000 7fa52752d700 1 --2- 172.21.15.173:0/3455689994 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa5201020b0 0x7fa5201976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:07.201 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:07.199+0000 7fa52752d700 1 --2- 172.21.15.173:0/3455689994 >> v2:172.21.15.72:6800/2923394687 conn(0x7fa50c06d990 0x7fa50c06fe50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:07.201 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:07.199+0000 7fa52752d700 1 --2- 172.21.15.173:0/3455689994 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa5201032b0 0x7fa520197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:07.201 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:07.199+0000 7fa52752d700 1 -- 172.21.15.173:0/3455689994 >> 172.21.15.173:0/3455689994 conn(0x7fa5200fd690 msgr2=0x7fa5201065b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:07.201 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:07.200+0000 7fa52752d700 1 -- 172.21.15.173:0/3455689994 shutdown_connections 2024-03-26T02:01:07.201 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:07.200+0000 7fa52752d700 1 -- 172.21.15.173:0/3455689994 wait complete. 2024-03-26T02:01:07.202 INFO:teuthology.orchestra.run.smithi173.stderr:pool 'datapool' created 2024-03-26T02:01:07.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.173:0/3455689994' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T02:01:07.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:07 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T02:01:07.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:07 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:07.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:07 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:07.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:07 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:07.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:07 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:07.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:07 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:07.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:07 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:07.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:07 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:01:07.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.173:0/3455689994' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T02:01:07.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:07 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T02:01:07.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:07 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:07.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:07 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:07.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:07 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:07.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:07 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:07.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:07 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:07.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:07 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:07.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:07 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:01:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.173:0/3455689994' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T02:01:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:07 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T02:01:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:07 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:07 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:07 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:07 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:07 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:07 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:07 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:01:07.873 DEBUG:teuthology.orchestra.run.smithi173:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- rbd pool init datapool 2024-03-26T02:01:08.113 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:01:08.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:08 smithi072 ceph-mon[41619]: Saving service rgw.foo.a spec with placement smithi072=foo.a;count:1 2024-03-26T02:01:08.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:08 smithi072 ceph-mon[41619]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T02:01:08.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T02:01:08.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:08 smithi072 ceph-mon[41619]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T02:01:08.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3766558542' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T02:01:08.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:08 smithi072 ceph-mon[30177]: Saving service rgw.foo.a spec with placement smithi072=foo.a;count:1 2024-03-26T02:01:08.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:08 smithi072 ceph-mon[30177]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T02:01:08.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T02:01:08.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:08 smithi072 ceph-mon[30177]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T02:01:08.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3766558542' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T02:01:08.475 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:08 smithi173 ceph-mon[33839]: Saving service rgw.foo.a spec with placement smithi072=foo.a;count:1 2024-03-26T02:01:08.475 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:08 smithi173 ceph-mon[33839]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T02:01:08.475 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T02:01:08.475 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:08 smithi173 ceph-mon[33839]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T02:01:08.475 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3766558542' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T02:01:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3766558542' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T02:01:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:09 smithi072 ceph-mon[30177]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T02:01:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:09 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:09 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:09 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:01:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:09 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:01:09.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3766558542' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T02:01:09.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:09 smithi072 ceph-mon[41619]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T02:01:09.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:09 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:09.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:09 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:09.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:09 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:01:09.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:09 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:01:09.423 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3766558542' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T02:01:09.423 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:09 smithi173 ceph-mon[33839]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T02:01:09.423 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:09 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:09.423 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:09 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:09.423 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:09 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:01:09.423 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:09 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:01:10.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:10 smithi072 ceph-mon[30177]: Checking dashboard <-> RGW credentials 2024-03-26T02:01:10.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:10 smithi072 ceph-mon[30177]: pgmap v191: 36 pgs: 12 creating+peering, 3 active+clean, 21 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:01:10.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:10 smithi072 ceph-mon[30177]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T02:01:10.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1842098834' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:01:10.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3862599162' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:01:10.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:01:10.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.173:0/2178752333' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T02:01:10.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T02:01:10.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:10 smithi072 ceph-mon[30177]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:01:10.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:10 smithi072 ceph-mon[41619]: Checking dashboard <-> RGW credentials 2024-03-26T02:01:10.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:10 smithi072 ceph-mon[41619]: pgmap v191: 36 pgs: 12 creating+peering, 3 active+clean, 21 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:01:10.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:10 smithi072 ceph-mon[41619]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T02:01:10.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1842098834' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:01:10.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3862599162' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:01:10.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:01:10.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.173:0/2178752333' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T02:01:10.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T02:01:10.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:10 smithi072 ceph-mon[41619]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:01:10.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:10 smithi173 ceph-mon[33839]: Checking dashboard <-> RGW credentials 2024-03-26T02:01:10.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:10 smithi173 ceph-mon[33839]: pgmap v191: 36 pgs: 12 creating+peering, 3 active+clean, 21 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:01:10.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:10 smithi173 ceph-mon[33839]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T02:01:10.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1842098834' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:01:10.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3862599162' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:01:10.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T02:01:10.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.173:0/2178752333' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T02:01:10.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T02:01:10.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:10 smithi173 ceph-mon[33839]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:01:11.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:11 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3862599162' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T02:01:11.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:11 smithi173 ceph-mon[33839]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T02:01:11.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:11 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T02:01:11.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:11 smithi173 ceph-mon[33839]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T02:01:11.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:11 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3862599162' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T02:01:11.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:11 smithi072 ceph-mon[30177]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T02:01:11.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:11 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T02:01:11.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:11 smithi072 ceph-mon[30177]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T02:01:11.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:11 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3862599162' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T02:01:11.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:11 smithi072 ceph-mon[41619]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T02:01:11.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:11 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T02:01:11.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:11 smithi072 ceph-mon[41619]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T02:01:12.544 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:12 smithi173 ceph-mon[33839]: pgmap v194: 68 pgs: 12 creating+peering, 34 active+clean, 22 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-26T02:01:12.544 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:12 smithi173 ceph-mon[33839]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T02:01:12.544 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:12 smithi173 ceph-mon[33839]: Cluster is now healthy 2024-03-26T02:01:12.544 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:12 smithi173 ceph-mon[33839]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T02:01:12.544 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3862599162' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:01:12.544 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1842098834' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:01:12.544 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:01:12.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:12 smithi072 ceph-mon[30177]: pgmap v194: 68 pgs: 12 creating+peering, 34 active+clean, 22 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-26T02:01:12.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:12 smithi072 ceph-mon[30177]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T02:01:12.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:12 smithi072 ceph-mon[30177]: Cluster is now healthy 2024-03-26T02:01:12.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:12 smithi072 ceph-mon[30177]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T02:01:12.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3862599162' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:01:12.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1842098834' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:01:12.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:01:12.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:12 smithi072 ceph-mon[41619]: pgmap v194: 68 pgs: 12 creating+peering, 34 active+clean, 22 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-03-26T02:01:12.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:12 smithi072 ceph-mon[41619]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T02:01:12.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:12 smithi072 ceph-mon[41619]: Cluster is now healthy 2024-03-26T02:01:12.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:12 smithi072 ceph-mon[41619]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T02:01:12.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3862599162' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:01:12.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1842098834' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:01:12.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T02:01:12.845 DEBUG:teuthology.orchestra.run.smithi173:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.173 --placement '1;smithi173=iscsi.a' 2024-03-26T02:01:13.081 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:01:13.472 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:13 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3862599162' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T02:01:13.472 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:13 smithi173 ceph-mon[33839]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T02:01:13.472 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:13 smithi173 ceph-mon[33839]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T02:01:13.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:13 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3862599162' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T02:01:13.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:13 smithi072 ceph-mon[41619]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T02:01:13.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:13 smithi072 ceph-mon[41619]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T02:01:13.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:13 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3862599162' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T02:01:13.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:13 smithi072 ceph-mon[30177]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T02:01:13.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:13 smithi072 ceph-mon[30177]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T02:01:14.373 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:14 smithi173 ceph-mon[33839]: pgmap v197: 100 pgs: 15 creating+peering, 57 active+clean, 28 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-26T02:01:14.373 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:14 smithi173 ceph-mon[33839]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T02:01:14.373 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:14 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3862599162' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:01:14.373 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:14 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1842098834' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:01:14.373 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:14 smithi173 ceph-mon[33839]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:01:14.639 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.639+0000 7f785f8d4700 1 -- 172.21.15.173:0/3632373407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78581003a0 msgr2=0x7f78581007c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:14.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.639+0000 7f785f8d4700 1 --2- 172.21.15.173:0/3632373407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78581003a0 0x7f78581007c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7854009990 tx=0x7f7854009ca0 comp rx=0 tx=0).stop 2024-03-26T02:01:14.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.639+0000 7f785f8d4700 1 -- 172.21.15.173:0/3632373407 shutdown_connections 2024-03-26T02:01:14.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.639+0000 7f785f8d4700 1 --2- 172.21.15.173:0/3632373407 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7858103dc0 0x7f785810a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:14.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.639+0000 7f785f8d4700 1 --2- 172.21.15.173:0/3632373407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f78581003a0 0x7f78581007c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:14.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.639+0000 7f785f8d4700 1 --2- 172.21.15.173:0/3632373407 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7858103420 0x7f7858103880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:14.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.639+0000 7f785f8d4700 1 -- 172.21.15.173:0/3632373407 >> 172.21.15.173:0/3632373407 conn(0x7f78580fbd20 msgr2=0x7f78580fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:14.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.639+0000 7f785f8d4700 1 -- 172.21.15.173:0/3632373407 shutdown_connections 2024-03-26T02:01:14.640 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.639+0000 7f785f8d4700 1 -- 172.21.15.173:0/3632373407 wait complete. 2024-03-26T02:01:14.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.639+0000 7f785f8d4700 1 Processor -- start 2024-03-26T02:01:14.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.640+0000 7f785f8d4700 1 -- start start 2024-03-26T02:01:14.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.640+0000 7f785f8d4700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78581003a0 0x7f7858102d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:14.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.640+0000 7f785f8d4700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7858103420 0x7f78581013d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:14.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.640+0000 7f785f8d4700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7858103dc0 0x7f7858101910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:14.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.640+0000 7f785f8d4700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7858102090 con 0x7f78581003a0 2024-03-26T02:01:14.642 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.640+0000 7f785f8d4700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7858102200 con 0x7f7858103dc0 2024-03-26T02:01:14.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.640+0000 7f785f8d4700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78581a1790 con 0x7f7858103420 2024-03-26T02:01:14.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.640+0000 7f785de71700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7858103dc0 0x7f7858101910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:14.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.640+0000 7f785de71700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7858103dc0 0x7f7858101910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:56190/0 (socket says 172.21.15.173:56190) 2024-03-26T02:01:14.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.640+0000 7f785de71700 1 -- 172.21.15.173:0/1940492278 learned_addr learned my addr 172.21.15.173:0/1940492278 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-26T02:01:14.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.640+0000 7f785ce6f700 1 --2- 172.21.15.173:0/1940492278 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7858103420 0x7f78581013d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:14.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.640+0000 7f785d670700 1 --2- 172.21.15.173:0/1940492278 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78581003a0 0x7f7858102d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:14.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.640+0000 7f785de71700 1 -- 172.21.15.173:0/1940492278 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7858103420 msgr2=0x7f78581013d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:14.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.640+0000 7f785de71700 1 --2- 172.21.15.173:0/1940492278 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7858103420 0x7f78581013d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:14.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.640+0000 7f785de71700 1 -- 172.21.15.173:0/1940492278 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78581003a0 msgr2=0x7f7858102d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:14.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.640+0000 7f785de71700 1 --2- 172.21.15.173:0/1940492278 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78581003a0 0x7f7858102d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:14.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.640+0000 7f785de71700 1 -- 172.21.15.173:0/1940492278 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78540095f0 con 0x7f7858103dc0 2024-03-26T02:01:14.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.640+0000 7f785de71700 1 --2- 172.21.15.173:0/1940492278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7858103dc0 0x7f7858101910 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7850007ee0 tx=0x7f785000d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:14.643 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.641+0000 7f784e7fc700 1 -- 172.21.15.173:0/1940492278 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7850017070 con 0x7f7858103dc0 2024-03-26T02:01:14.644 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.641+0000 7f784e7fc700 1 -- 172.21.15.173:0/1940492278 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f785000f040 con 0x7f7858103dc0 2024-03-26T02:01:14.644 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.641+0000 7f785f8d4700 1 -- 172.21.15.173:0/1940492278 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78581a1a70 con 0x7f7858103dc0 2024-03-26T02:01:14.644 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.641+0000 7f784e7fc700 1 -- 172.21.15.173:0/1940492278 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78500055a0 con 0x7f7858103dc0 2024-03-26T02:01:14.644 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.641+0000 7f785f8d4700 1 -- 172.21.15.173:0/1940492278 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78581a1f20 con 0x7f7858103dc0 2024-03-26T02:01:14.644 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.642+0000 7f785f8d4700 1 -- 172.21.15.173:0/1940492278 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7858109580 con 0x7f7858103dc0 2024-03-26T02:01:14.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.642+0000 7f784e7fc700 1 -- 172.21.15.173:0/1940492278 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 15) v1 ==== 91363+0+0 (secure 0 0 0) 0x7f785000db20 con 0x7f7858103dc0 2024-03-26T02:01:14.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.643+0000 7f784e7fc700 1 --2- 172.21.15.173:0/1940492278 >> v2:172.21.15.72:6800/2923394687 conn(0x7f784406da30 0x7f784406fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:14.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.643+0000 7f784e7fc700 1 -- 172.21.15.173:0/1940492278 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5926+0+0 (secure 0 0 0) 0x7f785008c870 con 0x7f7858103dc0 2024-03-26T02:01:14.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.643+0000 7f785d670700 1 --2- 172.21.15.173:0/1940492278 >> v2:172.21.15.72:6800/2923394687 conn(0x7f784406da30 0x7f784406fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:14.645 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.643+0000 7f785d670700 1 --2- 172.21.15.173:0/1940492278 >> v2:172.21.15.72:6800/2923394687 conn(0x7f784406da30 0x7f784406fef0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f78540036c0 tx=0x7f7854005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:14.648 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.648+0000 7f784e7fc700 1 -- 172.21.15.173:0/1940492278 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f78500562b0 con 0x7f7858103dc0 2024-03-26T02:01:14.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:14 smithi072 ceph-mon[30177]: pgmap v197: 100 pgs: 15 creating+peering, 57 active+clean, 28 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-26T02:01:14.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:14 smithi072 ceph-mon[30177]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T02:01:14.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:14 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3862599162' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:01:14.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:14 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1842098834' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:01:14.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:14 smithi072 ceph-mon[30177]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:01:14.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:14 smithi072 ceph-mon[41619]: pgmap v197: 100 pgs: 15 creating+peering, 57 active+clean, 28 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-26T02:01:14.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:14 smithi072 ceph-mon[41619]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T02:01:14.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:14 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3862599162' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:01:14.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:14 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1842098834' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:01:14.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:14 smithi072 ceph-mon[41619]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T02:01:14.854 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.853+0000 7f785f8d4700 1 -- 172.21.15.173:0/1940492278 --> v2:172.21.15.72:6800/2923394687 -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.173", "placement": "1;smithi173=iscsi.a", "target": ["mon-mgr", ""]}) v1 -- 0x7f78580624c0 con 0x7f784406da30 2024-03-26T02:01:14.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.861+0000 7f784e7fc700 1 -- 172.21.15.173:0/1940492278 <== mgr.14152 v2:172.21.15.72:6800/2923394687 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+35 (secure 0 0 0) 0x7f78580624c0 con 0x7f784406da30 2024-03-26T02:01:14.862 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled iscsi.datapool update... 2024-03-26T02:01:14.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.864+0000 7f785f8d4700 1 -- 172.21.15.173:0/1940492278 >> v2:172.21.15.72:6800/2923394687 conn(0x7f784406da30 msgr2=0x7f784406fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:14.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.864+0000 7f785f8d4700 1 --2- 172.21.15.173:0/1940492278 >> v2:172.21.15.72:6800/2923394687 conn(0x7f784406da30 0x7f784406fef0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f78540036c0 tx=0x7f7854005fb0 comp rx=0 tx=0).stop 2024-03-26T02:01:14.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.864+0000 7f785f8d4700 1 -- 172.21.15.173:0/1940492278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7858103dc0 msgr2=0x7f7858101910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:14.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.864+0000 7f785f8d4700 1 --2- 172.21.15.173:0/1940492278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7858103dc0 0x7f7858101910 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7850007ee0 tx=0x7f785000d3b0 comp rx=0 tx=0).stop 2024-03-26T02:01:14.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.864+0000 7f785f8d4700 1 -- 172.21.15.173:0/1940492278 shutdown_connections 2024-03-26T02:01:14.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.864+0000 7f785f8d4700 1 --2- 172.21.15.173:0/1940492278 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7858103dc0 0x7f7858101910 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:14.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.864+0000 7f785f8d4700 1 --2- 172.21.15.173:0/1940492278 >> v2:172.21.15.72:6800/2923394687 conn(0x7f784406da30 0x7f784406fef0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:14.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.864+0000 7f785f8d4700 1 --2- 172.21.15.173:0/1940492278 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f78581003a0 0x7f7858102d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:14.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.864+0000 7f785f8d4700 1 --2- 172.21.15.173:0/1940492278 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7858103420 0x7f78581013d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:14.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.864+0000 7f785f8d4700 1 -- 172.21.15.173:0/1940492278 >> 172.21.15.173:0/1940492278 conn(0x7f78580fbd20 msgr2=0x7f78581066c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:14.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.864+0000 7f785f8d4700 1 -- 172.21.15.173:0/1940492278 shutdown_connections 2024-03-26T02:01:14.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:14.864+0000 7f785f8d4700 1 -- 172.21.15.173:0/1940492278 wait complete. 2024-03-26T02:01:15.466 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3862599162' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T02:01:15.466 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T02:01:15.466 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:15 smithi173 ceph-mon[33839]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T02:01:15.466 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1842098834' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:01:15.466 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:01:15.466 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3862599162' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:01:15.466 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:15 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:15.500 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-26T02:01:15.500 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-26T02:01:15.501 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-26T02:01:15.534 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3862599162' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T02:01:15.534 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T02:01:15.535 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:15 smithi072 ceph-mon[30177]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T02:01:15.535 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1842098834' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:01:15.535 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:01:15.535 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3862599162' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:01:15.535 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:15 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:15.535 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3862599162' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T02:01:15.535 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T02:01:15.536 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:15 smithi072 ceph-mon[41619]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T02:01:15.536 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1842098834' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:01:15.536 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:01:15.536 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3862599162' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T02:01:15.536 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:15 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:15.539 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-26T02:01:15.539 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-26T02:01:15.571 DEBUG:teuthology.orchestra.run.smithi173:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@iscsi.iscsi.a.service 2024-03-26T02:01:15.613 INFO:tasks.cephadm:Adding prometheus.a on smithi173 2024-03-26T02:01:15.614 DEBUG:teuthology.orchestra.run.smithi173:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph orch apply prometheus '1;smithi173=a' 2024-03-26T02:01:15.897 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:01:16.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:16 smithi173 ceph-mon[33839]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.173", "placement": "1;smithi173=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:16.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:16 smithi173 ceph-mon[33839]: Saving service iscsi.datapool spec with placement smithi173=iscsi.a;count:1 2024-03-26T02:01:16.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:16 smithi173 ceph-mon[33839]: pgmap v200: 132 pgs: 15 creating+peering, 84 active+clean, 33 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1023 B/s wr, 2 op/s 2024-03-26T02:01:16.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:16 smithi173 ceph-mon[33839]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T02:01:16.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:16 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3862599162' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T02:01:16.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:16 smithi173 ceph-mon[33839]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T02:01:16.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:16 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:16.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:16 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:16.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:16 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:16.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:16 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:01:16.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:16 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T02:01:16.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:16 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:01:16.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:16 smithi173 ceph-mon[33839]: Deploying daemon iscsi.iscsi.a on smithi173 2024-03-26T02:01:16.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:16 smithi072 ceph-mon[30177]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.173", "placement": "1;smithi173=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:16.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:16 smithi072 ceph-mon[30177]: Saving service iscsi.datapool spec with placement smithi173=iscsi.a;count:1 2024-03-26T02:01:16.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:16 smithi072 ceph-mon[30177]: pgmap v200: 132 pgs: 15 creating+peering, 84 active+clean, 33 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1023 B/s wr, 2 op/s 2024-03-26T02:01:16.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:16 smithi072 ceph-mon[30177]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T02:01:16.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:16 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3862599162' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T02:01:16.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:16 smithi072 ceph-mon[30177]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T02:01:16.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:16 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:16.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:16 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:16.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:16 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:16.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:16 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:01:16.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:16 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T02:01:16.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:16 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:01:16.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:16 smithi072 ceph-mon[30177]: Deploying daemon iscsi.iscsi.a on smithi173 2024-03-26T02:01:16.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:16 smithi072 ceph-mon[41619]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.173", "placement": "1;smithi173=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:16.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:16 smithi072 ceph-mon[41619]: Saving service iscsi.datapool spec with placement smithi173=iscsi.a;count:1 2024-03-26T02:01:16.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:16 smithi072 ceph-mon[41619]: pgmap v200: 132 pgs: 15 creating+peering, 84 active+clean, 33 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 1023 B/s wr, 2 op/s 2024-03-26T02:01:16.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:16 smithi072 ceph-mon[41619]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T02:01:16.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:16 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3862599162' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T02:01:16.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:16 smithi072 ceph-mon[41619]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T02:01:16.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:16 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:16.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:16 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:16.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:16 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:16.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:16 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T02:01:16.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:16 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T02:01:16.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:16 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:01:16.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:16 smithi072 ceph-mon[41619]: Deploying daemon iscsi.iscsi.a on smithi173 2024-03-26T02:01:17.355 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:17 smithi173 ceph-mon[33839]: pgmap v202: 132 pgs: 11 creating+peering, 108 active+clean, 13 unknown; 580 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 5.1 KiB/s wr, 187 op/s 2024-03-26T02:01:17.718 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:17 smithi072 ceph-mon[30177]: pgmap v202: 132 pgs: 11 creating+peering, 108 active+clean, 13 unknown; 580 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 5.1 KiB/s wr, 187 op/s 2024-03-26T02:01:17.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:17 smithi072 ceph-mon[41619]: pgmap v202: 132 pgs: 11 creating+peering, 108 active+clean, 13 unknown; 580 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s rd, 5.1 KiB/s wr, 187 op/s 2024-03-26T02:01:18.698 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.695+0000 7f85bb0ab700 1 -- 172.21.15.173:0/1841039478 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85b410a130 msgr2=0x7f85b410a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:18.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.695+0000 7f85bb0ab700 1 --2- 172.21.15.173:0/1841039478 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85b410a130 0x7f85b410a590 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f85a000aaa0 tx=0x7f85a000adb0 comp rx=0 tx=0).stop 2024-03-26T02:01:18.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.695+0000 7f85bb0ab700 1 -- 172.21.15.173:0/1841039478 shutdown_connections 2024-03-26T02:01:18.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.695+0000 7f85bb0ab700 1 --2- 172.21.15.173:0/1841039478 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85b410aad0 0x7f85b4111380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:18.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.695+0000 7f85bb0ab700 1 --2- 172.21.15.173:0/1841039478 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85b41094c0 0x7f85b41098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:18.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.695+0000 7f85bb0ab700 1 --2- 172.21.15.173:0/1841039478 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85b410a130 0x7f85b410a590 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:18.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.695+0000 7f85bb0ab700 1 -- 172.21.15.173:0/1841039478 >> 172.21.15.173:0/1841039478 conn(0x7f85b406ef50 msgr2=0x7f85b4071390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:18.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.695+0000 7f85bb0ab700 1 -- 172.21.15.173:0/1841039478 shutdown_connections 2024-03-26T02:01:18.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.695+0000 7f85bb0ab700 1 -- 172.21.15.173:0/1841039478 wait complete. 2024-03-26T02:01:18.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.696+0000 7f85bb0ab700 1 Processor -- start 2024-03-26T02:01:18.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.696+0000 7f85bb0ab700 1 -- start start 2024-03-26T02:01:18.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.696+0000 7f85bb0ab700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85b41094c0 0x7f85b419fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:18.699 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.696+0000 7f85bb0ab700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85b410a130 0x7f85b41a0420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:18.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.696+0000 7f85bb0ab700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85b410aad0 0x7f85b41a74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:18.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.697+0000 7f85bb0ab700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b41a7a70 con 0x7f85b410aad0 2024-03-26T02:01:18.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.697+0000 7f85bb0ab700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b41a7bb0 con 0x7f85b41094c0 2024-03-26T02:01:18.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.697+0000 7f85bb0ab700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b41a7d40 con 0x7f85b410a130 2024-03-26T02:01:18.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.697+0000 7f85b8e47700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85b41094c0 0x7f85b419fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:18.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.697+0000 7f85b8e47700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85b41094c0 0x7f85b419fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:52652/0 (socket says 172.21.15.173:52652) 2024-03-26T02:01:18.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.697+0000 7f85b8e47700 1 -- 172.21.15.173:0/937924528 learned_addr learned my addr 172.21.15.173:0/937924528 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-26T02:01:18.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.697+0000 7f85b3fff700 1 --2- 172.21.15.173:0/937924528 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85b410a130 0x7f85b41a0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:18.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.697+0000 7f85b9648700 1 --2- 172.21.15.173:0/937924528 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85b410aad0 0x7f85b41a74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:18.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.697+0000 7f85b8e47700 1 -- 172.21.15.173:0/937924528 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85b410a130 msgr2=0x7f85b41a0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:18.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.697+0000 7f85b8e47700 1 --2- 172.21.15.173:0/937924528 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85b410a130 0x7f85b41a0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:18.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.697+0000 7f85b8e47700 1 -- 172.21.15.173:0/937924528 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85b410aad0 msgr2=0x7f85b41a74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:18.700 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.697+0000 7f85b8e47700 1 --2- 172.21.15.173:0/937924528 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85b410aad0 0x7f85b41a74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:18.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.697+0000 7f85b8e47700 1 -- 172.21.15.173:0/937924528 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a000a700 con 0x7f85b41094c0 2024-03-26T02:01:18.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.697+0000 7f85b9648700 1 --2- 172.21.15.173:0/937924528 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85b410aad0 0x7f85b41a74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T02:01:18.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.697+0000 7f85b8e47700 1 --2- 172.21.15.173:0/937924528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85b41094c0 0x7f85b419fee0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f85a800c970 tx=0x7f85a800cd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:18.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.697+0000 7f85b1ffb700 1 -- 172.21.15.173:0/937924528 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a8004220 con 0x7f85b41094c0 2024-03-26T02:01:18.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.697+0000 7f85bb0ab700 1 -- 172.21.15.173:0/937924528 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b41a7fc0 con 0x7f85b41094c0 2024-03-26T02:01:18.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.697+0000 7f85b1ffb700 1 -- 172.21.15.173:0/937924528 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f85a8007d50 con 0x7f85b41094c0 2024-03-26T02:01:18.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.697+0000 7f85b1ffb700 1 -- 172.21.15.173:0/937924528 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a8003d10 con 0x7f85b41094c0 2024-03-26T02:01:18.701 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.698+0000 7f85bb0ab700 1 -- 172.21.15.173:0/937924528 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b41a8480 con 0x7f85b41094c0 2024-03-26T02:01:18.702 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.699+0000 7f85bb0ab700 1 -- 172.21.15.173:0/937924528 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85b4068d40 con 0x7f85b41094c0 2024-03-26T02:01:18.702 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.699+0000 7f85b1ffb700 1 -- 172.21.15.173:0/937924528 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 15) v1 ==== 91363+0+0 (secure 0 0 0) 0x7f85a80078a0 con 0x7f85b41094c0 2024-03-26T02:01:18.702 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.700+0000 7f85b1ffb700 1 --2- 172.21.15.173:0/937924528 >> v2:172.21.15.72:6800/2923394687 conn(0x7f85a406d990 0x7f85a406fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:18.702 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.700+0000 7f85b1ffb700 1 -- 172.21.15.173:0/937924528 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5942+0+0 (secure 0 0 0) 0x7f85a808cd90 con 0x7f85b41094c0 2024-03-26T02:01:18.702 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.700+0000 7f85b3fff700 1 --2- 172.21.15.173:0/937924528 >> v2:172.21.15.72:6800/2923394687 conn(0x7f85a406d990 0x7f85a406fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:18.702 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.701+0000 7f85b3fff700 1 --2- 172.21.15.173:0/937924528 >> v2:172.21.15.72:6800/2923394687 conn(0x7f85a406d990 0x7f85a406fe50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f85a000c6d0 tx=0x7f85a0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:18.705 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.705+0000 7f85b1ffb700 1 -- 172.21.15.173:0/937924528 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f85a80567c0 con 0x7f85b41094c0 2024-03-26T02:01:18.929 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.928+0000 7f85bb0ab700 1 -- 172.21.15.173:0/937924528 --> v2:172.21.15.72:6800/2923394687 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f85b4110c90 con 0x7f85a406d990 2024-03-26T02:01:18.937 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.936+0000 7f85b1ffb700 1 -- 172.21.15.173:0/937924528 <== mgr.14152 v2:172.21.15.72:6800/2923394687 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f85b4110c90 con 0x7f85a406d990 2024-03-26T02:01:18.937 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled prometheus update... 2024-03-26T02:01:18.940 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.939+0000 7f85bb0ab700 1 -- 172.21.15.173:0/937924528 >> v2:172.21.15.72:6800/2923394687 conn(0x7f85a406d990 msgr2=0x7f85a406fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:18.940 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.939+0000 7f85bb0ab700 1 --2- 172.21.15.173:0/937924528 >> v2:172.21.15.72:6800/2923394687 conn(0x7f85a406d990 0x7f85a406fe50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f85a000c6d0 tx=0x7f85a0005fb0 comp rx=0 tx=0).stop 2024-03-26T02:01:18.940 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.939+0000 7f85bb0ab700 1 -- 172.21.15.173:0/937924528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85b41094c0 msgr2=0x7f85b419fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:18.940 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.939+0000 7f85bb0ab700 1 --2- 172.21.15.173:0/937924528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85b41094c0 0x7f85b419fee0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f85a800c970 tx=0x7f85a800cd30 comp rx=0 tx=0).stop 2024-03-26T02:01:18.940 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.939+0000 7f85bb0ab700 1 -- 172.21.15.173:0/937924528 shutdown_connections 2024-03-26T02:01:18.940 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.939+0000 7f85bb0ab700 1 --2- 172.21.15.173:0/937924528 >> v2:172.21.15.72:6800/2923394687 conn(0x7f85a406d990 0x7f85a406fe50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:18.940 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.939+0000 7f85bb0ab700 1 --2- 172.21.15.173:0/937924528 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f85b410aad0 0x7f85b41a74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:18.940 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.939+0000 7f85bb0ab700 1 --2- 172.21.15.173:0/937924528 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85b41094c0 0x7f85b419fee0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:18.940 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.939+0000 7f85bb0ab700 1 --2- 172.21.15.173:0/937924528 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f85b410a130 0x7f85b41a0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:18.940 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.939+0000 7f85bb0ab700 1 -- 172.21.15.173:0/937924528 >> 172.21.15.173:0/937924528 conn(0x7f85b406ef50 msgr2=0x7f85b410f570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:18.940 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.939+0000 7f85bb0ab700 1 -- 172.21.15.173:0/937924528 shutdown_connections 2024-03-26T02:01:18.941 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:18.939+0000 7f85bb0ab700 1 -- 172.21.15.173:0/937924528 wait complete. 2024-03-26T02:01:19.272 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:01:19 smithi173 systemd[1]: Starting Ceph iscsi.iscsi.a for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:01:19.556 DEBUG:teuthology.orchestra.run.smithi173:prometheus.a> sudo journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@prometheus.a.service 2024-03-26T02:01:19.558 INFO:tasks.cephadm:Adding node-exporter.a on smithi072 2024-03-26T02:01:19.558 INFO:tasks.cephadm:Adding node-exporter.b on smithi173 2024-03-26T02:01:19.558 DEBUG:teuthology.orchestra.run.smithi173:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph orch apply node-exporter '2;smithi072=a;smithi173=b' 2024-03-26T02:01:19.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:01:19 smithi173 podman[62815]: 2024-03-26 02:01:19.772675629 +0000 UTC m=+0.113773321 container create 3a58dadfedfafb6e9796d22a7868e28822528a3dae2dfb5b070b68c02a0e413f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-26T02:01:19.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:01:19 smithi173 podman[62815]: 2024-03-26 02:01:19.680600545 +0000 UTC m=+0.021698248 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:01:20.130 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:01:20.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:19 smithi072 ceph-mon[30177]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:20.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:19 smithi072 ceph-mon[30177]: Saving service prometheus spec with placement smithi173=a;count:1 2024-03-26T02:01:20.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:19 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:20.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:19 smithi072 ceph-mon[30177]: pgmap v203: 132 pgs: 6 creating+peering, 126 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 119 KiB/s rd, 5.8 KiB/s wr, 266 op/s 2024-03-26T02:01:20.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:19 smithi072 ceph-mon[41619]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:20.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:19 smithi072 ceph-mon[41619]: Saving service prometheus spec with placement smithi173=a;count:1 2024-03-26T02:01:20.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:19 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:20.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:19 smithi072 ceph-mon[41619]: pgmap v203: 132 pgs: 6 creating+peering, 126 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 119 KiB/s rd, 5.8 KiB/s wr, 266 op/s 2024-03-26T02:01:20.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:19 smithi173 ceph-mon[33839]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:20.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:19 smithi173 ceph-mon[33839]: Saving service prometheus spec with placement smithi173=a;count:1 2024-03-26T02:01:20.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:19 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:20.243 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:19 smithi173 ceph-mon[33839]: pgmap v203: 132 pgs: 6 creating+peering, 126 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 119 KiB/s rd, 5.8 KiB/s wr, 266 op/s 2024-03-26T02:01:20.244 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:01:19 smithi173 podman[62815]: 2024-03-26 02:01:19.966778425 +0000 UTC m=+0.307876115 container init 3a58dadfedfafb6e9796d22a7868e28822528a3dae2dfb5b070b68c02a0e413f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-26T02:01:20.244 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:01:19 smithi173 podman[62815]: 2024-03-26 02:01:19.972107415 +0000 UTC m=+0.313205115 container start 3a58dadfedfafb6e9796d22a7868e28822528a3dae2dfb5b070b68c02a0e413f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-26T02:01:20.244 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:01:20 smithi173 bash[62815]: 3a58dadfedfafb6e9796d22a7868e28822528a3dae2dfb5b070b68c02a0e413f 2024-03-26T02:01:20.244 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:01:20 smithi173 systemd[1]: Started Ceph iscsi.iscsi.a for c86fbdd4-eb13-11ee-95d0-87774f69a715. 2024-03-26T02:01:20.502 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:01:20 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug Started the configuration object watcher 2024-03-26T02:01:20.502 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:01:20 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug Checking for config object changes every 1s 2024-03-26T02:01:20.503 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:01:20 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug Processing osd blocklist entries for this node 2024-03-26T02:01:20.898 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:01:20 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug Reading the configuration object to update local LIO configuration 2024-03-26T02:01:20.898 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:01:20 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug Configuration does not have an entry for this host(smithi173.front.sepia.ceph.com) - nothing to define to LIO 2024-03-26T02:01:20.898 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:01:20 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-26T02:01:21.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:21 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:21.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:21 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:21.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:21 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:21.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:21 smithi072 ceph-mon[41619]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T02:01:21.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:21 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:21.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:21 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:01:21.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.173:0/350819173' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T02:01:21.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:21 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:21.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:21 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:21.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:21 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:21.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:21 smithi072 ceph-mon[30177]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T02:01:21.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:21 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:21.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:21 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:01:21.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.173:0/350819173' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T02:01:21.485 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:21 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:21.485 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:21 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:21.485 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:21 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:21.485 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:21 smithi173 ceph-mon[33839]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T02:01:21.485 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:21 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:21.485 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:21 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:01:21.486 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.173:0/350819173' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T02:01:22.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.320+0000 7fe83fc04700 1 -- 172.21.15.173:0/1117798571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe838103c50 msgr2=0x7fe83810a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:22.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.320+0000 7fe83fc04700 1 --2- 172.21.15.173:0/1117798571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe838103c50 0x7fe83810a500 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe83000b0d0 tx=0x7fe83000b3e0 comp rx=0 tx=0).stop 2024-03-26T02:01:22.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.320+0000 7fe83fc04700 1 -- 172.21.15.173:0/1117798571 shutdown_connections 2024-03-26T02:01:22.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.320+0000 7fe83fc04700 1 --2- 172.21.15.173:0/1117798571 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe838103c50 0x7fe83810a500 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:22.321 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.320+0000 7fe83fc04700 1 --2- 172.21.15.173:0/1117798571 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe8381020b0 0x7fe8381024d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:22.322 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.320+0000 7fe83fc04700 1 --2- 172.21.15.173:0/1117798571 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8381032b0 0x7fe838103710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:22.322 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.320+0000 7fe83fc04700 1 -- 172.21.15.173:0/1117798571 >> 172.21.15.173:0/1117798571 conn(0x7fe8380fd690 msgr2=0x7fe8380ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:22.322 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.320+0000 7fe83fc04700 1 -- 172.21.15.173:0/1117798571 shutdown_connections 2024-03-26T02:01:22.322 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.320+0000 7fe83fc04700 1 -- 172.21.15.173:0/1117798571 wait complete. 2024-03-26T02:01:22.322 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.320+0000 7fe83fc04700 1 Processor -- start 2024-03-26T02:01:22.322 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.320+0000 7fe83fc04700 1 -- start start 2024-03-26T02:01:22.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.321+0000 7fe83fc04700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8381020b0 0x7fe838199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:22.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.321+0000 7fe83fc04700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8381032b0 0x7fe838199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:22.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.321+0000 7fe83fc04700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe838103c50 0x7fe8381a1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:22.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.321+0000 7fe83fc04700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8381a15e0 con 0x7fe8381032b0 2024-03-26T02:01:22.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.321+0000 7fe83fc04700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8381a1720 con 0x7fe8381020b0 2024-03-26T02:01:22.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.321+0000 7fe83fc04700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8381a18b0 con 0x7fe838103c50 2024-03-26T02:01:22.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.321+0000 7fe83d9a0700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8381020b0 0x7fe838199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:22.324 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.321+0000 7fe83d9a0700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8381020b0 0x7fe838199a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:52726/0 (socket says 172.21.15.173:52726) 2024-03-26T02:01:22.325 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.321+0000 7fe83d9a0700 1 -- 172.21.15.173:0/119234515 learned_addr learned my addr 172.21.15.173:0/119234515 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-26T02:01:22.325 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.321+0000 7fe83d19f700 1 --2- 172.21.15.173:0/119234515 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8381032b0 0x7fe838199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:22.325 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.321+0000 7fe83e1a1700 1 --2- 172.21.15.173:0/119234515 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe838103c50 0x7fe8381a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:22.325 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.321+0000 7fe83e1a1700 1 -- 172.21.15.173:0/119234515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8381020b0 msgr2=0x7fe838199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:22.325 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.321+0000 7fe83e1a1700 1 --2- 172.21.15.173:0/119234515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8381020b0 0x7fe838199a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:22.325 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.321+0000 7fe83e1a1700 1 -- 172.21.15.173:0/119234515 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8381032b0 msgr2=0x7fe838199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:22.325 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.321+0000 7fe83e1a1700 1 --2- 172.21.15.173:0/119234515 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8381032b0 0x7fe838199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:22.325 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.321+0000 7fe83e1a1700 1 -- 172.21.15.173:0/119234515 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe830009d00 con 0x7fe838103c50 2024-03-26T02:01:22.325 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.321+0000 7fe83d19f700 1 --2- 172.21.15.173:0/119234515 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8381032b0 0x7fe838199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:01:22.325 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.321+0000 7fe83d9a0700 1 --2- 172.21.15.173:0/119234515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8381020b0 0x7fe838199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:01:22.325 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.321+0000 7fe83e1a1700 1 --2- 172.21.15.173:0/119234515 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe838103c50 0x7fe8381a1010 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe83000b0d0 tx=0x7fe830007870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:22.325 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.322+0000 7fe82effd700 1 -- 172.21.15.173:0/119234515 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83000e040 con 0x7fe838103c50 2024-03-26T02:01:22.325 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.322+0000 7fe83fc04700 1 -- 172.21.15.173:0/119234515 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8381a1b30 con 0x7fe838103c50 2024-03-26T02:01:22.325 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.322+0000 7fe82effd700 1 -- 172.21.15.173:0/119234515 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fe83001d930 con 0x7fe838103c50 2024-03-26T02:01:22.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.322+0000 7fe82effd700 1 -- 172.21.15.173:0/119234515 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83001b8e0 con 0x7fe838103c50 2024-03-26T02:01:22.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.322+0000 7fe83fc04700 1 -- 172.21.15.173:0/119234515 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8381a1fa0 con 0x7fe838103c50 2024-03-26T02:01:22.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.322+0000 7fe83fc04700 1 -- 172.21.15.173:0/119234515 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe838068d40 con 0x7fe838103c50 2024-03-26T02:01:22.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.323+0000 7fe82effd700 1 -- 172.21.15.173:0/119234515 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 16) v1 ==== 91414+0+0 (secure 0 0 0) 0x7fe83001d480 con 0x7fe838103c50 2024-03-26T02:01:22.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.323+0000 7fe82effd700 1 --2- 172.21.15.173:0/119234515 >> v2:172.21.15.72:6800/2923394687 conn(0x7fe82406da20 0x7fe82406fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:22.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.324+0000 7fe82effd700 1 -- 172.21.15.173:0/119234515 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5942+0+0 (secure 0 0 0) 0x7fe83008ed20 con 0x7fe838103c50 2024-03-26T02:01:22.326 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.324+0000 7fe83d9a0700 1 --2- 172.21.15.173:0/119234515 >> v2:172.21.15.72:6800/2923394687 conn(0x7fe82406da20 0x7fe82406fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:22.327 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.324+0000 7fe83d9a0700 1 --2- 172.21.15.173:0/119234515 >> v2:172.21.15.72:6800/2923394687 conn(0x7fe82406da20 0x7fe82406fee0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe8340105c0 tx=0x7fe83400b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:22.329 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.329+0000 7fe82effd700 1 -- 172.21.15.173:0/119234515 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe830093070 con 0x7fe838103c50 2024-03-26T02:01:22.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:22 smithi072 ceph-mon[30177]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 4.6 KiB/s wr, 203 op/s 2024-03-26T02:01:22.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:22 smithi072 ceph-mon[30177]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T02:01:22.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:22 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:22.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:22 smithi072 ceph-mon[41619]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 4.6 KiB/s wr, 203 op/s 2024-03-26T02:01:22.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:22 smithi072 ceph-mon[41619]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T02:01:22.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:22 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:22.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:22 smithi173 ceph-mon[33839]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s rd, 4.6 KiB/s wr, 203 op/s 2024-03-26T02:01:22.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:22 smithi173 ceph-mon[33839]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T02:01:22.515 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:22 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:22.539 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.538+0000 7fe83fc04700 1 -- 172.21.15.173:0/119234515 --> v2:172.21.15.72:6800/2923394687 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi072=a;smithi173=b", "target": ["mon-mgr", ""]}) v1 -- 0x7fe8380759d0 con 0x7fe82406da20 2024-03-26T02:01:22.548 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.548+0000 7fe82effd700 1 -- 172.21.15.173:0/119234515 <== mgr.14152 v2:172.21.15.72:6800/2923394687 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fe8380759d0 con 0x7fe82406da20 2024-03-26T02:01:22.548 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled node-exporter update... 2024-03-26T02:01:22.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.550+0000 7fe83fc04700 1 -- 172.21.15.173:0/119234515 >> v2:172.21.15.72:6800/2923394687 conn(0x7fe82406da20 msgr2=0x7fe82406fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:22.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.550+0000 7fe83fc04700 1 --2- 172.21.15.173:0/119234515 >> v2:172.21.15.72:6800/2923394687 conn(0x7fe82406da20 0x7fe82406fee0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe8340105c0 tx=0x7fe83400b4a0 comp rx=0 tx=0).stop 2024-03-26T02:01:22.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.550+0000 7fe83fc04700 1 -- 172.21.15.173:0/119234515 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe838103c50 msgr2=0x7fe8381a1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:22.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.550+0000 7fe83fc04700 1 --2- 172.21.15.173:0/119234515 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe838103c50 0x7fe8381a1010 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe83000b0d0 tx=0x7fe830007870 comp rx=0 tx=0).stop 2024-03-26T02:01:22.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.550+0000 7fe83fc04700 1 -- 172.21.15.173:0/119234515 shutdown_connections 2024-03-26T02:01:22.551 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.550+0000 7fe83fc04700 1 --2- 172.21.15.173:0/119234515 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe838103c50 0x7fe8381a1010 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:22.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.550+0000 7fe83fc04700 1 --2- 172.21.15.173:0/119234515 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe8381020b0 0x7fe838199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:22.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.550+0000 7fe83fc04700 1 --2- 172.21.15.173:0/119234515 >> v2:172.21.15.72:6800/2923394687 conn(0x7fe82406da20 0x7fe82406fee0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:22.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.550+0000 7fe83fc04700 1 --2- 172.21.15.173:0/119234515 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe8381032b0 0x7fe838199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:22.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.551+0000 7fe83fc04700 1 -- 172.21.15.173:0/119234515 >> 172.21.15.173:0/119234515 conn(0x7fe8380fd690 msgr2=0x7fe8380742b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:22.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.551+0000 7fe83fc04700 1 -- 172.21.15.173:0/119234515 shutdown_connections 2024-03-26T02:01:22.552 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:22.551+0000 7fe83fc04700 1 -- 172.21.15.173:0/119234515 wait complete. 2024-03-26T02:01:23.196 DEBUG:teuthology.orchestra.run.smithi072:node-exporter.a> sudo journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@node-exporter.a.service 2024-03-26T02:01:23.199 DEBUG:teuthology.orchestra.run.smithi173:node-exporter.b> sudo journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@node-exporter.b.service 2024-03-26T02:01:23.202 INFO:tasks.cephadm:Adding alertmanager.a on smithi072 2024-03-26T02:01:23.202 DEBUG:teuthology.orchestra.run.smithi173:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph orch apply alertmanager '1;smithi072=a' 2024-03-26T02:01:23.538 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:01:23.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:23 smithi173 ceph-mon[33839]: from='client.24431 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi072=a;smithi173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:23.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:23 smithi173 ceph-mon[33839]: Saving service node-exporter spec with placement smithi072=a;smithi173=b;count:2 2024-03-26T02:01:23.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:23 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:23.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:23 smithi173 ceph-mon[33839]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 3.9 KiB/s wr, 180 op/s 2024-03-26T02:01:23.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:23 smithi072 ceph-mon[30177]: from='client.24431 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi072=a;smithi173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:23.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:23 smithi072 ceph-mon[30177]: Saving service node-exporter spec with placement smithi072=a;smithi173=b;count:2 2024-03-26T02:01:23.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:23 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:23.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:23 smithi072 ceph-mon[30177]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 3.9 KiB/s wr, 180 op/s 2024-03-26T02:01:23.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:23 smithi072 ceph-mon[41619]: from='client.24431 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi072=a;smithi173=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:23.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:23 smithi072 ceph-mon[41619]: Saving service node-exporter spec with placement smithi072=a;smithi173=b;count:2 2024-03-26T02:01:23.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:23 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:23.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:23 smithi072 ceph-mon[41619]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 81 KiB/s rd, 3.9 KiB/s wr, 180 op/s 2024-03-26T02:01:25.018 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:24 smithi173 ceph-mon[33839]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T02:01:25.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:24 smithi072 ceph-mon[30177]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T02:01:25.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:24 smithi072 ceph-mon[41619]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T02:01:25.524 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.449+0000 7f9695890700 1 -- 172.21.15.173:0/554642439 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9690102c00 msgr2=0x7f9690109490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:25.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.449+0000 7f9695890700 1 --2- 172.21.15.173:0/554642439 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9690102c00 0x7f9690109490 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f968800b0a0 tx=0x7f968800b3b0 comp rx=0 tx=0).stop 2024-03-26T02:01:25.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.449+0000 7f9695890700 1 -- 172.21.15.173:0/554642439 shutdown_connections 2024-03-26T02:01:25.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.449+0000 7f9695890700 1 --2- 172.21.15.173:0/554642439 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9690102c00 0x7f9690109490 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:25.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.449+0000 7f9695890700 1 --2- 172.21.15.173:0/554642439 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9690101520 0x7f9690101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:25.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.449+0000 7f9695890700 1 --2- 172.21.15.173:0/554642439 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9690102240 0x7f96901026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:25.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.449+0000 7f9695890700 1 -- 172.21.15.173:0/554642439 >> 172.21.15.173:0/554642439 conn(0x7f96900fbd20 msgr2=0x7f96900fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:25.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.449+0000 7f9695890700 1 -- 172.21.15.173:0/554642439 shutdown_connections 2024-03-26T02:01:25.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.449+0000 7f9695890700 1 -- 172.21.15.173:0/554642439 wait complete. 2024-03-26T02:01:25.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.450+0000 7f9695890700 1 Processor -- start 2024-03-26T02:01:25.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.450+0000 7f9695890700 1 -- start start 2024-03-26T02:01:25.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.450+0000 7f9695890700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9690101520 0x7f9690100be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:25.525 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.450+0000 7f9695890700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9690102240 0x7f969019c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:25.526 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.450+0000 7f9695890700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9690102c00 0x7f969019ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:25.526 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.450+0000 7f9695890700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969019f3b0 con 0x7f9690102240 2024-03-26T02:01:25.526 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.450+0000 7f9695890700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f969019f520 con 0x7f9690101520 2024-03-26T02:01:25.526 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.450+0000 7f9695890700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f969019f6e0 con 0x7f9690102c00 2024-03-26T02:01:25.526 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.450+0000 7f968effd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9690101520 0x7f9690100be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:25.526 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.450+0000 7f968effd700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9690101520 0x7f9690100be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:52746/0 (socket says 172.21.15.173:52746) 2024-03-26T02:01:25.526 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.450+0000 7f968effd700 1 -- 172.21.15.173:0/1114192396 learned_addr learned my addr 172.21.15.173:0/1114192396 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-26T02:01:25.526 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.450+0000 7f968e7fc700 1 --2- 172.21.15.173:0/1114192396 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9690102240 0x7f969019c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:25.526 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.450+0000 7f968f7fe700 1 --2- 172.21.15.173:0/1114192396 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9690102c00 0x7f969019ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:25.526 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.451+0000 7f968effd700 1 -- 172.21.15.173:0/1114192396 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9690102c00 msgr2=0x7f969019ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:25.526 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.451+0000 7f968effd700 1 --2- 172.21.15.173:0/1114192396 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9690102c00 0x7f969019ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:25.526 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.451+0000 7f968effd700 1 -- 172.21.15.173:0/1114192396 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9690102240 msgr2=0x7f969019c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:25.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.451+0000 7f968effd700 1 --2- 172.21.15.173:0/1114192396 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9690102240 0x7f969019c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:25.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.451+0000 7f968effd700 1 -- 172.21.15.173:0/1114192396 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9688009d00 con 0x7f9690101520 2024-03-26T02:01:25.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.451+0000 7f968effd700 1 --2- 172.21.15.173:0/1114192396 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9690101520 0x7f9690100be0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f968000e970 tx=0x7f968000ec80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:25.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.451+0000 7f969488e700 1 -- 172.21.15.173:0/1114192396 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9680009790 con 0x7f9690101520 2024-03-26T02:01:25.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.451+0000 7f9695890700 1 -- 172.21.15.173:0/1114192396 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f969019f9c0 con 0x7f9690101520 2024-03-26T02:01:25.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.451+0000 7f969488e700 1 -- 172.21.15.173:0/1114192396 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f968000c7e0 con 0x7f9690101520 2024-03-26T02:01:25.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.451+0000 7f969488e700 1 -- 172.21.15.173:0/1114192396 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9680003bd0 con 0x7f9690101520 2024-03-26T02:01:25.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.451+0000 7f9695890700 1 -- 172.21.15.173:0/1114192396 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f969019fec0 con 0x7f9690101520 2024-03-26T02:01:25.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.452+0000 7f9695890700 1 -- 172.21.15.173:0/1114192396 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9690108330 con 0x7f9690101520 2024-03-26T02:01:25.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.452+0000 7f969488e700 1 -- 172.21.15.173:0/1114192396 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 91414+0+0 (secure 0 0 0) 0x7f9680003d70 con 0x7f9690101520 2024-03-26T02:01:25.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.453+0000 7f969488e700 1 --2- 172.21.15.173:0/1114192396 >> v2:172.21.15.72:6800/2923394687 conn(0x7f967806da20 0x7f967806fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:25.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.453+0000 7f969488e700 1 -- 172.21.15.173:0/1114192396 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5942+0+0 (secure 0 0 0) 0x7f9680014070 con 0x7f9690101520 2024-03-26T02:01:25.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.453+0000 7f968e7fc700 1 --2- 172.21.15.173:0/1114192396 >> v2:172.21.15.72:6800/2923394687 conn(0x7f967806da20 0x7f967806fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:25.527 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.453+0000 7f968e7fc700 1 --2- 172.21.15.173:0/1114192396 >> v2:172.21.15.72:6800/2923394687 conn(0x7f967806da20 0x7f967806fee0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9690101160 tx=0x7f968400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:25.528 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.457+0000 7f969488e700 1 -- 172.21.15.173:0/1114192396 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f9680056db0 con 0x7f9690101520 2024-03-26T02:01:25.669 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.668+0000 7f9695890700 1 -- 172.21.15.173:0/1114192396 --> v2:172.21.15.72:6800/2923394687 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi072=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f96900624c0 con 0x7f967806da20 2024-03-26T02:01:25.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.676+0000 7f969488e700 1 -- 172.21.15.173:0/1114192396 <== mgr.14152 v2:172.21.15.72:6800/2923394687 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f96900624c0 con 0x7f967806da20 2024-03-26T02:01:25.676 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled alertmanager update... 2024-03-26T02:01:25.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.678+0000 7f9695890700 1 -- 172.21.15.173:0/1114192396 >> v2:172.21.15.72:6800/2923394687 conn(0x7f967806da20 msgr2=0x7f967806fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:25.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.678+0000 7f9695890700 1 --2- 172.21.15.173:0/1114192396 >> v2:172.21.15.72:6800/2923394687 conn(0x7f967806da20 0x7f967806fee0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9690101160 tx=0x7f968400b410 comp rx=0 tx=0).stop 2024-03-26T02:01:25.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.678+0000 7f9695890700 1 -- 172.21.15.173:0/1114192396 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9690101520 msgr2=0x7f9690100be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:25.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.678+0000 7f9695890700 1 --2- 172.21.15.173:0/1114192396 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9690101520 0x7f9690100be0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f968000e970 tx=0x7f968000ec80 comp rx=0 tx=0).stop 2024-03-26T02:01:25.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.678+0000 7f9695890700 1 -- 172.21.15.173:0/1114192396 shutdown_connections 2024-03-26T02:01:25.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.678+0000 7f9695890700 1 --2- 172.21.15.173:0/1114192396 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9690102c00 0x7f969019ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:25.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.678+0000 7f9695890700 1 --2- 172.21.15.173:0/1114192396 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9690101520 0x7f9690100be0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:25.679 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.678+0000 7f9695890700 1 --2- 172.21.15.173:0/1114192396 >> v2:172.21.15.72:6800/2923394687 conn(0x7f967806da20 0x7f967806fee0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:25.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.678+0000 7f9695890700 1 --2- 172.21.15.173:0/1114192396 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9690102240 0x7f969019c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:25.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.678+0000 7f9695890700 1 -- 172.21.15.173:0/1114192396 >> 172.21.15.173:0/1114192396 conn(0x7f96900fbd20 msgr2=0x7f9690105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:25.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.678+0000 7f9695890700 1 -- 172.21.15.173:0/1114192396 shutdown_connections 2024-03-26T02:01:25.680 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:25.678+0000 7f9695890700 1 -- 172.21.15.173:0/1114192396 wait complete. 2024-03-26T02:01:26.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:25 smithi173 ceph-mon[33839]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 3.6 KiB/s wr, 163 op/s 2024-03-26T02:01:26.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:25 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:26.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:25 smithi072 ceph-mon[30177]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 3.6 KiB/s wr, 163 op/s 2024-03-26T02:01:26.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:25 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:26.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:25 smithi072 ceph-mon[41619]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s rd, 3.6 KiB/s wr, 163 op/s 2024-03-26T02:01:26.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:25 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:26.425 DEBUG:teuthology.orchestra.run.smithi072:alertmanager.a> sudo journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@alertmanager.a.service 2024-03-26T02:01:26.427 INFO:tasks.cephadm:Adding grafana.a on smithi173 2024-03-26T02:01:26.428 DEBUG:teuthology.orchestra.run.smithi173:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph orch apply grafana '1;smithi173=a' 2024-03-26T02:01:26.679 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:01:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:26 smithi173 ceph-mon[33839]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi072=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:26 smithi173 ceph-mon[33839]: Saving service alertmanager spec with placement smithi072=a;count:1 2024-03-26T02:01:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:26 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:26 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:26 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:01:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:26 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:01:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:26 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:26 smithi173 ceph-mon[33839]: Deploying daemon prometheus.a on smithi173 2024-03-26T02:01:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:26 smithi072 ceph-mon[30177]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi072=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:26 smithi072 ceph-mon[30177]: Saving service alertmanager spec with placement smithi072=a;count:1 2024-03-26T02:01:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:26 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:26 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:26 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:01:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:26 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:01:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:26 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:26 smithi072 ceph-mon[30177]: Deploying daemon prometheus.a on smithi173 2024-03-26T02:01:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:26 smithi072 ceph-mon[41619]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi072=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:26 smithi072 ceph-mon[41619]: Saving service alertmanager spec with placement smithi072=a;count:1 2024-03-26T02:01:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:26 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:26 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:26 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:01:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:26 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:01:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:26 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:26 smithi072 ceph-mon[41619]: Deploying daemon prometheus.a on smithi173 2024-03-26T02:01:28.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:27 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-26T02:01:28.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:27 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-26T02:01:28.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:27 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-26T02:01:28.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:27 smithi072 ceph-mon[30177]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 1.4 KiB/s wr, 70 op/s 2024-03-26T02:01:28.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:27 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:28.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:27 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-26T02:01:28.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:27 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-26T02:01:28.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:27 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-26T02:01:28.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:27 smithi072 ceph-mon[41619]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 1.4 KiB/s wr, 70 op/s 2024-03-26T02:01:28.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:27 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:28.312 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:27 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]: dispatch 2024-03-26T02:01:28.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:27 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-26T02:01:28.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:27 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-03-26T02:01:28.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:27 smithi173 ceph-mon[33839]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 1.4 KiB/s wr, 70 op/s 2024-03-26T02:01:28.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:27 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:28.949 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:28 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-03-26T02:01:28.949 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:28 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-26T02:01:28.949 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:28 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-03-26T02:01:28.949 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:28 smithi173 ceph-mon[33839]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T02:01:29.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:28 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-03-26T02:01:29.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:28 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-26T02:01:29.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:28 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-03-26T02:01:29.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:28 smithi072 ceph-mon[30177]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T02:01:29.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:28 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [1, 2]}]': finished 2024-03-26T02:01:29.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:28 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-26T02:01:29.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:28 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-03-26T02:01:29.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:28 smithi072 ceph-mon[41619]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T02:01:29.259 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.222+0000 7ff13b1be700 1 -- 172.21.15.173:0/4232514419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff134105460 msgr2=0x7ff134107850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:29.259 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.222+0000 7ff13b1be700 1 --2- 172.21.15.173:0/4232514419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff134105460 0x7ff134107850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff120009990 tx=0x7ff120009ca0 comp rx=0 tx=0).stop 2024-03-26T02:01:29.259 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.222+0000 7ff13b1be700 1 -- 172.21.15.173:0/4232514419 shutdown_connections 2024-03-26T02:01:29.259 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.222+0000 7ff13b1be700 1 --2- 172.21.15.173:0/4232514419 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff134107d90 0x7ff13410a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:29.259 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.222+0000 7ff13b1be700 1 --2- 172.21.15.173:0/4232514419 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff134102b30 0x7ff134104f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:29.259 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.222+0000 7ff13b1be700 1 --2- 172.21.15.173:0/4232514419 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff134105460 0x7ff134107850 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:29.259 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.222+0000 7ff13b1be700 1 -- 172.21.15.173:0/4232514419 >> 172.21.15.173:0/4232514419 conn(0x7ff1340fc540 msgr2=0x7ff1340fe980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:29.259 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.222+0000 7ff13b1be700 1 -- 172.21.15.173:0/4232514419 shutdown_connections 2024-03-26T02:01:29.259 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.222+0000 7ff13b1be700 1 -- 172.21.15.173:0/4232514419 wait complete. 2024-03-26T02:01:29.259 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.222+0000 7ff13b1be700 1 Processor -- start 2024-03-26T02:01:29.259 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.222+0000 7ff13b1be700 1 -- start start 2024-03-26T02:01:29.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.223+0000 7ff13b1be700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff134102b30 0x7ff134196cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:29.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.223+0000 7ff13b1be700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff134105460 0x7ff134195360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:29.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.223+0000 7ff13b1be700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff134107d90 0x7ff1341958a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:29.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.223+0000 7ff13b1be700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff134196020 con 0x7ff134102b30 2024-03-26T02:01:29.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.223+0000 7ff13b1be700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff134196190 con 0x7ff134105460 2024-03-26T02:01:29.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.223+0000 7ff13b1be700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1341a5ba0 con 0x7ff134107d90 2024-03-26T02:01:29.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.223+0000 7ff133fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff134105460 0x7ff134195360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:29.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.223+0000 7ff133fff700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff134105460 0x7ff134195360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:36136/0 (socket says 172.21.15.173:36136) 2024-03-26T02:01:29.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.223+0000 7ff133fff700 1 -- 172.21.15.173:0/4143653526 learned_addr learned my addr 172.21.15.173:0/4143653526 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-26T02:01:29.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.223+0000 7ff138f5a700 1 --2- 172.21.15.173:0/4143653526 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff134102b30 0x7ff134196cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:29.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.223+0000 7ff13975b700 1 --2- 172.21.15.173:0/4143653526 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff134107d90 0x7ff1341958a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:29.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.223+0000 7ff133fff700 1 -- 172.21.15.173:0/4143653526 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff134107d90 msgr2=0x7ff1341958a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:29.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.223+0000 7ff133fff700 1 --2- 172.21.15.173:0/4143653526 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff134107d90 0x7ff1341958a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:29.260 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.223+0000 7ff133fff700 1 -- 172.21.15.173:0/4143653526 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff134102b30 msgr2=0x7ff134196cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:29.261 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.223+0000 7ff133fff700 1 --2- 172.21.15.173:0/4143653526 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff134102b30 0x7ff134196cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:29.261 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.223+0000 7ff133fff700 1 -- 172.21.15.173:0/4143653526 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1200095f0 con 0x7ff134105460 2024-03-26T02:01:29.261 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.223+0000 7ff133fff700 1 --2- 172.21.15.173:0/4143653526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff134105460 0x7ff134195360 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff120006010 tx=0x7ff120004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:29.261 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.223+0000 7ff131ffb700 1 -- 172.21.15.173:0/4143653526 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12001d070 con 0x7ff134105460 2024-03-26T02:01:29.261 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.223+0000 7ff13b1be700 1 -- 172.21.15.173:0/4143653526 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1341a5e20 con 0x7ff134105460 2024-03-26T02:01:29.261 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.223+0000 7ff131ffb700 1 -- 172.21.15.173:0/4143653526 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7ff120004c90 con 0x7ff134105460 2024-03-26T02:01:29.261 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.223+0000 7ff131ffb700 1 -- 172.21.15.173:0/4143653526 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff120003b20 con 0x7ff134105460 2024-03-26T02:01:29.261 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.224+0000 7ff13b1be700 1 -- 172.21.15.173:0/4143653526 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1341a6230 con 0x7ff134105460 2024-03-26T02:01:29.261 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.225+0000 7ff13b1be700 1 -- 172.21.15.173:0/4143653526 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff134109990 con 0x7ff134105460 2024-03-26T02:01:29.261 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.225+0000 7ff131ffb700 1 -- 172.21.15.173:0/4143653526 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 91414+0+0 (secure 0 0 0) 0x7ff120003ca0 con 0x7ff134105460 2024-03-26T02:01:29.261 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.225+0000 7ff131ffb700 1 --2- 172.21.15.173:0/4143653526 >> v2:172.21.15.72:6800/2923394687 conn(0x7ff12406da20 0x7ff12406fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:29.261 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.225+0000 7ff131ffb700 1 -- 172.21.15.173:0/4143653526 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6001+0+0 (secure 0 0 0) 0x7ff12008e240 con 0x7ff134105460 2024-03-26T02:01:29.261 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.226+0000 7ff138f5a700 1 --2- 172.21.15.173:0/4143653526 >> v2:172.21.15.72:6800/2923394687 conn(0x7ff12406da20 0x7ff12406fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:29.261 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.226+0000 7ff138f5a700 1 --2- 172.21.15.173:0/4143653526 >> v2:172.21.15.72:6800/2923394687 conn(0x7ff12406da20 0x7ff12406fee0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff128005950 tx=0x7ff12800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:29.261 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.230+0000 7ff131ffb700 1 -- 172.21.15.173:0/4143653526 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7ff120057c10 con 0x7ff134105460 2024-03-26T02:01:29.440 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.439+0000 7ff13b1be700 1 -- 172.21.15.173:0/4143653526 --> v2:172.21.15.72:6800/2923394687 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1340624c0 con 0x7ff12406da20 2024-03-26T02:01:29.447 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.447+0000 7ff131ffb700 1 -- 172.21.15.173:0/4143653526 <== mgr.14152 v2:172.21.15.72:6800/2923394687 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7ff1340624c0 con 0x7ff12406da20 2024-03-26T02:01:29.448 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled grafana update... 2024-03-26T02:01:29.450 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.449+0000 7ff13b1be700 1 -- 172.21.15.173:0/4143653526 >> v2:172.21.15.72:6800/2923394687 conn(0x7ff12406da20 msgr2=0x7ff12406fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:29.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.449+0000 7ff13b1be700 1 --2- 172.21.15.173:0/4143653526 >> v2:172.21.15.72:6800/2923394687 conn(0x7ff12406da20 0x7ff12406fee0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ff128005950 tx=0x7ff12800b410 comp rx=0 tx=0).stop 2024-03-26T02:01:29.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.449+0000 7ff13b1be700 1 -- 172.21.15.173:0/4143653526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff134105460 msgr2=0x7ff134195360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:29.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.449+0000 7ff13b1be700 1 --2- 172.21.15.173:0/4143653526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff134105460 0x7ff134195360 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff120006010 tx=0x7ff120004730 comp rx=0 tx=0).stop 2024-03-26T02:01:29.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.449+0000 7ff13b1be700 1 -- 172.21.15.173:0/4143653526 shutdown_connections 2024-03-26T02:01:29.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.449+0000 7ff13b1be700 1 --2- 172.21.15.173:0/4143653526 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7ff134107d90 0x7ff1341958a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:29.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.449+0000 7ff13b1be700 1 --2- 172.21.15.173:0/4143653526 >> v2:172.21.15.72:6800/2923394687 conn(0x7ff12406da20 0x7ff12406fee0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:29.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.449+0000 7ff13b1be700 1 --2- 172.21.15.173:0/4143653526 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7ff134102b30 0x7ff134196cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:29.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.449+0000 7ff13b1be700 1 --2- 172.21.15.173:0/4143653526 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff134105460 0x7ff134195360 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:29.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.449+0000 7ff13b1be700 1 -- 172.21.15.173:0/4143653526 >> 172.21.15.173:0/4143653526 conn(0x7ff1340fc540 msgr2=0x7ff1341037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:29.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.449+0000 7ff13b1be700 1 -- 172.21.15.173:0/4143653526 shutdown_connections 2024-03-26T02:01:29.451 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:29.449+0000 7ff13b1be700 1 -- 172.21.15.173:0/4143653526 wait complete. 2024-03-26T02:01:30.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:29 smithi072 ceph-mon[30177]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T02:01:30.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:29 smithi072 ceph-mon[30177]: pgmap v211: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 170 B/s wr, 2 op/s 2024-03-26T02:01:30.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:29 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:30.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:29 smithi072 ceph-mon[41619]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T02:01:30.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:29 smithi072 ceph-mon[41619]: pgmap v211: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 170 B/s wr, 2 op/s 2024-03-26T02:01:30.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:29 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:30.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:29 smithi173 ceph-mon[33839]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T02:01:30.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:29 smithi173 ceph-mon[33839]: pgmap v211: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 170 B/s wr, 2 op/s 2024-03-26T02:01:30.265 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:29 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:30.375 DEBUG:teuthology.orchestra.run.smithi173:grafana.a> sudo journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@grafana.a.service 2024-03-26T02:01:30.378 INFO:tasks.cephadm:Setting up client nodes... 2024-03-26T02:01:30.378 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T02:01:30.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:01:30 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:01:30.627 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:01:31.005 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:30 smithi072 ceph-mon[30177]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:31.005 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:30 smithi072 ceph-mon[30177]: Saving service grafana spec with placement smithi173=a;count:1 2024-03-26T02:01:31.005 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:30 smithi072 ceph-mon[41619]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:31.005 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:30 smithi072 ceph-mon[41619]: Saving service grafana spec with placement smithi173=a;count:1 2024-03-26T02:01:31.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:30 smithi173 ceph-mon[33839]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi173=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T02:01:31.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:30 smithi173 ceph-mon[33839]: Saving service grafana spec with placement smithi173=a;count:1 2024-03-26T02:01:32.096 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:31 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:01:32.097 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:31 smithi072 ceph-mon[30177]: pgmap v212: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 139 B/s wr, 2 op/s; 44 B/s, 2 objects/s recovering 2024-03-26T02:01:32.097 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:31 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:01:32.097 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:31 smithi072 ceph-mon[41619]: pgmap v212: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 139 B/s wr, 2 op/s; 44 B/s, 2 objects/s recovering 2024-03-26T02:01:32.166 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:31 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:01:32.166 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:31 smithi173 ceph-mon[33839]: pgmap v212: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 139 B/s wr, 2 op/s; 44 B/s, 2 objects/s recovering 2024-03-26T02:01:32.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.371+0000 7fea4ec9b700 1 -- 172.21.15.72:0/4040633901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea48103c70 msgr2=0x7fea4810a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:32.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.371+0000 7fea4ec9b700 1 --2- 172.21.15.72:0/4040633901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea48103c70 0x7fea4810a520 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fea38009920 tx=0x7fea38009c30 comp rx=0 tx=0).stop 2024-03-26T02:01:32.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.371+0000 7fea4ec9b700 1 -- 172.21.15.72:0/4040633901 shutdown_connections 2024-03-26T02:01:32.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.371+0000 7fea4ec9b700 1 --2- 172.21.15.72:0/4040633901 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea48103c70 0x7fea4810a520 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:32.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.371+0000 7fea4ec9b700 1 --2- 172.21.15.72:0/4040633901 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea481020d0 0x7fea481024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:32.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.371+0000 7fea4ec9b700 1 --2- 172.21.15.72:0/4040633901 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fea481032d0 0x7fea48103730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:32.373 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.371+0000 7fea4ec9b700 1 -- 172.21.15.72:0/4040633901 >> 172.21.15.72:0/4040633901 conn(0x7fea480fd690 msgr2=0x7fea480ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:32.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.371+0000 7fea4ec9b700 1 -- 172.21.15.72:0/4040633901 shutdown_connections 2024-03-26T02:01:32.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.371+0000 7fea4ec9b700 1 -- 172.21.15.72:0/4040633901 wait complete. 2024-03-26T02:01:32.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.371+0000 7fea4ec9b700 1 Processor -- start 2024-03-26T02:01:32.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.371+0000 7fea4ec9b700 1 -- start start 2024-03-26T02:01:32.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.371+0000 7fea4ec9b700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea481020d0 0x7fea48075e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:32.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.371+0000 7fea4ec9b700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fea481032d0 0x7fea48074470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:32.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.372+0000 7fea4ec9b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea48103c70 0x7fea480749b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:32.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.372+0000 7fea4ec9b700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea48075130 con 0x7fea481020d0 2024-03-26T02:01:32.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.372+0000 7fea4ec9b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea480752a0 con 0x7fea48103c70 2024-03-26T02:01:32.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.372+0000 7fea4ec9b700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea481a17a0 con 0x7fea481032d0 2024-03-26T02:01:32.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.372+0000 7fea4ca37700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea481020d0 0x7fea48075e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:32.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.372+0000 7fea47fff700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fea481032d0 0x7fea48074470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:32.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.372+0000 7fea4ca37700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea481020d0 0x7fea48075e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43050/0 (socket says 172.21.15.72:43050) 2024-03-26T02:01:32.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.372+0000 7fea4ca37700 1 -- 172.21.15.72:0/496980869 learned_addr learned my addr 172.21.15.72:0/496980869 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:01:32.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.372+0000 7fea4d238700 1 --2- 172.21.15.72:0/496980869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea48103c70 0x7fea480749b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:32.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.372+0000 7fea4ca37700 1 -- 172.21.15.72:0/496980869 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fea481032d0 msgr2=0x7fea48074470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:32.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.372+0000 7fea4ca37700 1 --2- 172.21.15.72:0/496980869 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fea481032d0 0x7fea48074470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:32.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.372+0000 7fea4ca37700 1 -- 172.21.15.72:0/496980869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea48103c70 msgr2=0x7fea480749b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:32.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.372+0000 7fea4ca37700 1 --2- 172.21.15.72:0/496980869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea48103c70 0x7fea480749b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:32.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.372+0000 7fea4ca37700 1 -- 172.21.15.72:0/496980869 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea3c0095f0 con 0x7fea481020d0 2024-03-26T02:01:32.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.372+0000 7fea4ca37700 1 --2- 172.21.15.72:0/496980869 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea481020d0 0x7fea48075e20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fea40007f10 tx=0x7fea4000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:32.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.372+0000 7fea45ffb700 1 -- 172.21.15.72:0/496980869 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea40017070 con 0x7fea481020d0 2024-03-26T02:01:32.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.372+0000 7fea45ffb700 1 -- 172.21.15.72:0/496980869 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fea4000f040 con 0x7fea481020d0 2024-03-26T02:01:32.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.372+0000 7fea4ec9b700 1 -- 172.21.15.72:0/496980869 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea38009520 con 0x7fea481020d0 2024-03-26T02:01:32.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.372+0000 7fea45ffb700 1 -- 172.21.15.72:0/496980869 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea400044d0 con 0x7fea481020d0 2024-03-26T02:01:32.375 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.372+0000 7fea4ec9b700 1 -- 172.21.15.72:0/496980869 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea481a1e40 con 0x7fea481020d0 2024-03-26T02:01:32.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.373+0000 7fea4ec9b700 1 -- 172.21.15.72:0/496980869 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea481093c0 con 0x7fea481020d0 2024-03-26T02:01:32.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.374+0000 7fea45ffb700 1 -- 172.21.15.72:0/496980869 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) v1 ==== 91414+0+0 (secure 0 0 0) 0x7fea40003be0 con 0x7fea481020d0 2024-03-26T02:01:32.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.374+0000 7fea45ffb700 1 --2- 172.21.15.72:0/496980869 >> v2:172.21.15.72:6800/2923394687 conn(0x7fea3006dac0 0x7fea3006ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:32.376 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.374+0000 7fea47fff700 1 --2- 172.21.15.72:0/496980869 >> v2:172.21.15.72:6800/2923394687 conn(0x7fea3006dac0 0x7fea3006ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:32.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.374+0000 7fea45ffb700 1 -- 172.21.15.72:0/496980869 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6001+0+0 (secure 0 0 0) 0x7fea4008c8f0 con 0x7fea481020d0 2024-03-26T02:01:32.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.375+0000 7fea47fff700 1 --2- 172.21.15.72:0/496980869 >> v2:172.21.15.72:6800/2923394687 conn(0x7fea3006dac0 0x7fea3006ff80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fea3c005fd0 tx=0x7fea3c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:32.379 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.379+0000 7fea45ffb700 1 -- 172.21.15.72:0/496980869 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fea40056260 con 0x7fea481020d0 2024-03-26T02:01:32.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.657+0000 7fea4ec9b700 1 -- 172.21.15.72:0/496980869 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fea48050590 con 0x7fea481020d0 2024-03-26T02:01:32.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.661+0000 7fea45ffb700 1 -- 172.21.15.72:0/496980869 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7fea40059920 con 0x7fea481020d0 2024-03-26T02:01:32.662 INFO:teuthology.orchestra.run.smithi072.stdout:[client.0] 2024-03-26T02:01:32.662 INFO:teuthology.orchestra.run.smithi072.stdout: key = AQB8LAJm9uNFJxAASlIBqWydWqKM9/ig91J2hA== 2024-03-26T02:01:32.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.664+0000 7fea4ec9b700 1 -- 172.21.15.72:0/496980869 >> v2:172.21.15.72:6800/2923394687 conn(0x7fea3006dac0 msgr2=0x7fea3006ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:32.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.664+0000 7fea4ec9b700 1 --2- 172.21.15.72:0/496980869 >> v2:172.21.15.72:6800/2923394687 conn(0x7fea3006dac0 0x7fea3006ff80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fea3c005fd0 tx=0x7fea3c009310 comp rx=0 tx=0).stop 2024-03-26T02:01:32.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.664+0000 7fea4ec9b700 1 -- 172.21.15.72:0/496980869 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea481020d0 msgr2=0x7fea48075e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:32.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.664+0000 7fea4ec9b700 1 --2- 172.21.15.72:0/496980869 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea481020d0 0x7fea48075e20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fea40007f10 tx=0x7fea4000d3b0 comp rx=0 tx=0).stop 2024-03-26T02:01:32.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.664+0000 7fea4ec9b700 1 -- 172.21.15.72:0/496980869 shutdown_connections 2024-03-26T02:01:32.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.664+0000 7fea4ec9b700 1 --2- 172.21.15.72:0/496980869 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea48103c70 0x7fea480749b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:32.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.664+0000 7fea4ec9b700 1 --2- 172.21.15.72:0/496980869 >> v2:172.21.15.72:6800/2923394687 conn(0x7fea3006dac0 0x7fea3006ff80 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:32.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.664+0000 7fea4ec9b700 1 --2- 172.21.15.72:0/496980869 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea481020d0 0x7fea48075e20 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:32.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.664+0000 7fea4ec9b700 1 --2- 172.21.15.72:0/496980869 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fea481032d0 0x7fea48074470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:32.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.664+0000 7fea4ec9b700 1 -- 172.21.15.72:0/496980869 >> 172.21.15.72:0/496980869 conn(0x7fea480fd690 msgr2=0x7fea48106500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:32.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.664+0000 7fea4ec9b700 1 -- 172.21.15.72:0/496980869 shutdown_connections 2024-03-26T02:01:32.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:32.664+0000 7fea4ec9b700 1 -- 172.21.15.72:0/496980869 wait complete. 2024-03-26T02:01:33.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/496980869' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:01:33.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/496980869' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:01:33.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/496980869' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:01:33.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/496980869' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:01:33.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/496980869' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:01:33.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/496980869' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:01:33.355 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-26T02:01:33.355 DEBUG:teuthology.orchestra.run.smithi072:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-26T02:01:33.355 DEBUG:teuthology.orchestra.run.smithi072:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-26T02:01:33.398 DEBUG:teuthology.orchestra.run.smithi173:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T02:01:34.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:33 smithi072 ceph-mon[30177]: pgmap v213: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 41 B/s, 1 objects/s recovering 2024-03-26T02:01:34.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:33 smithi072 ceph-mon[41619]: pgmap v213: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 41 B/s, 1 objects/s recovering 2024-03-26T02:01:34.242 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T02:01:34.312 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:33 smithi173 ceph-mon[33839]: pgmap v213: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 41 B/s, 1 objects/s recovering 2024-03-26T02:01:36.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:36 smithi173 ceph-mon[33839]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 2 objects/s recovering 2024-03-26T02:01:36.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:36 smithi072 ceph-mon[30177]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 2 objects/s recovering 2024-03-26T02:01:36.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:36 smithi072 ceph-mon[41619]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 2 objects/s recovering 2024-03-26T02:01:37.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.072+0000 7fba80c61700 1 -- 172.21.15.173:0/3460279661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba7c1020d0 msgr2=0x7fba7c1024f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:37.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.072+0000 7fba80c61700 1 --2- 172.21.15.173:0/3460279661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba7c1020d0 0x7fba7c1024f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fba6c0099c0 tx=0x7fba6c009cd0 comp rx=0 tx=0).stop 2024-03-26T02:01:37.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.072+0000 7fba80c61700 1 -- 172.21.15.173:0/3460279661 shutdown_connections 2024-03-26T02:01:37.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.072+0000 7fba80c61700 1 --2- 172.21.15.173:0/3460279661 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba7c103c70 0x7fba7c10a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:37.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.072+0000 7fba80c61700 1 --2- 172.21.15.173:0/3460279661 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba7c1020d0 0x7fba7c1024f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:37.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.072+0000 7fba80c61700 1 --2- 172.21.15.173:0/3460279661 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fba7c1032d0 0x7fba7c103730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:37.074 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.072+0000 7fba80c61700 1 -- 172.21.15.173:0/3460279661 >> 172.21.15.173:0/3460279661 conn(0x7fba7c0fd690 msgr2=0x7fba7c0ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:37.074 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.072+0000 7fba80c61700 1 -- 172.21.15.173:0/3460279661 shutdown_connections 2024-03-26T02:01:37.074 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.072+0000 7fba80c61700 1 -- 172.21.15.173:0/3460279661 wait complete. 2024-03-26T02:01:37.077 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.073+0000 7fba80c61700 1 Processor -- start 2024-03-26T02:01:37.077 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.073+0000 7fba80c61700 1 -- start start 2024-03-26T02:01:37.077 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.073+0000 7fba80c61700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba7c1020d0 0x7fba7c075e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:37.077 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.073+0000 7fba80c61700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba7c1032d0 0x7fba7c074470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:37.078 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.073+0000 7fba80c61700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fba7c103c70 0x7fba7c0749b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:37.078 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.073+0000 7fba80c61700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c075130 con 0x7fba7c1032d0 2024-03-26T02:01:37.078 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.073+0000 7fba80c61700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c0752a0 con 0x7fba7c1020d0 2024-03-26T02:01:37.078 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.074+0000 7fba80c61700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba7c1a1610 con 0x7fba7c103c70 2024-03-26T02:01:37.078 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.074+0000 7fba7a59c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba7c1020d0 0x7fba7c075e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:37.078 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.074+0000 7fba7a59c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba7c1020d0 0x7fba7c075e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:36150/0 (socket says 172.21.15.173:36150) 2024-03-26T02:01:37.078 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.074+0000 7fba7a59c700 1 -- 172.21.15.173:0/3390387344 learned_addr learned my addr 172.21.15.173:0/3390387344 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-03-26T02:01:37.078 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.074+0000 7fba73fff700 1 --2- 172.21.15.173:0/3390387344 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba7c1032d0 0x7fba7c074470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:37.078 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.074+0000 7fba7ad9d700 1 --2- 172.21.15.173:0/3390387344 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fba7c103c70 0x7fba7c0749b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:37.078 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.074+0000 7fba7a59c700 1 -- 172.21.15.173:0/3390387344 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fba7c103c70 msgr2=0x7fba7c0749b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:37.078 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.074+0000 7fba7a59c700 1 --2- 172.21.15.173:0/3390387344 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fba7c103c70 0x7fba7c0749b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:37.078 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.074+0000 7fba7a59c700 1 -- 172.21.15.173:0/3390387344 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba7c1032d0 msgr2=0x7fba7c074470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:37.078 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.074+0000 7fba7a59c700 1 --2- 172.21.15.173:0/3390387344 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba7c1032d0 0x7fba7c074470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:37.079 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.074+0000 7fba7a59c700 1 -- 172.21.15.173:0/3390387344 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba6c0095f0 con 0x7fba7c1020d0 2024-03-26T02:01:37.079 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.074+0000 7fba7a59c700 1 --2- 172.21.15.173:0/3390387344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba7c1020d0 0x7fba7c075e20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fba6c009f50 tx=0x7fba6c0047a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:37.079 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.074+0000 7fba737fe700 1 -- 172.21.15.173:0/3390387344 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6c01d070 con 0x7fba7c1020d0 2024-03-26T02:01:37.079 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.074+0000 7fba737fe700 1 -- 172.21.15.173:0/3390387344 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fba6c004120 con 0x7fba7c1020d0 2024-03-26T02:01:37.079 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.074+0000 7fba80c61700 1 -- 172.21.15.173:0/3390387344 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba7c1a1890 con 0x7fba7c1020d0 2024-03-26T02:01:37.079 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.074+0000 7fba737fe700 1 -- 172.21.15.173:0/3390387344 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba6c022ea0 con 0x7fba7c1020d0 2024-03-26T02:01:37.079 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.074+0000 7fba80c61700 1 -- 172.21.15.173:0/3390387344 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba7c1a1ca0 con 0x7fba7c1020d0 2024-03-26T02:01:37.079 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.075+0000 7fba80c61700 1 -- 172.21.15.173:0/3390387344 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba7c1093c0 con 0x7fba7c1020d0 2024-03-26T02:01:37.079 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.076+0000 7fba737fe700 1 -- 172.21.15.173:0/3390387344 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 16) v1 ==== 91414+0+0 (secure 0 0 0) 0x7fba6c003c70 con 0x7fba7c1020d0 2024-03-26T02:01:37.079 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.076+0000 7fba737fe700 1 --2- 172.21.15.173:0/3390387344 >> v2:172.21.15.72:6800/2923394687 conn(0x7fba5c06dac0 0x7fba5c06ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:37.079 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.076+0000 7fba737fe700 1 -- 172.21.15.173:0/3390387344 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6001+0+0 (secure 0 0 0) 0x7fba6c08e650 con 0x7fba7c1020d0 2024-03-26T02:01:37.079 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.077+0000 7fba73fff700 1 --2- 172.21.15.173:0/3390387344 >> v2:172.21.15.72:6800/2923394687 conn(0x7fba5c06dac0 0x7fba5c06ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:37.080 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.077+0000 7fba73fff700 1 --2- 172.21.15.173:0/3390387344 >> v2:172.21.15.72:6800/2923394687 conn(0x7fba5c06dac0 0x7fba5c06ff80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fba6400ac50 tx=0x7fba6400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:37.081 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.081+0000 7fba737fe700 1 -- 172.21.15.173:0/3390387344 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fba6c057ff0 con 0x7fba7c1020d0 2024-03-26T02:01:37.359 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.358+0000 7fba80c61700 1 -- 172.21.15.173:0/3390387344 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fba7c10d340 con 0x7fba7c1020d0 2024-03-26T02:01:37.366 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.366+0000 7fba737fe700 1 -- 172.21.15.173:0/3390387344 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7fba6c05b6b0 con 0x7fba7c1020d0 2024-03-26T02:01:37.366 INFO:teuthology.orchestra.run.smithi173.stdout:[client.1] 2024-03-26T02:01:37.366 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQCBLAJmFIyCFRAAXPEAF+0w+VjDzT6KcqfF7A== 2024-03-26T02:01:37.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.368+0000 7fba80c61700 1 -- 172.21.15.173:0/3390387344 >> v2:172.21.15.72:6800/2923394687 conn(0x7fba5c06dac0 msgr2=0x7fba5c06ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:37.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.368+0000 7fba80c61700 1 --2- 172.21.15.173:0/3390387344 >> v2:172.21.15.72:6800/2923394687 conn(0x7fba5c06dac0 0x7fba5c06ff80 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fba6400ac50 tx=0x7fba6400a380 comp rx=0 tx=0).stop 2024-03-26T02:01:37.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.368+0000 7fba80c61700 1 -- 172.21.15.173:0/3390387344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba7c1020d0 msgr2=0x7fba7c075e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:37.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.368+0000 7fba80c61700 1 --2- 172.21.15.173:0/3390387344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba7c1020d0 0x7fba7c075e20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fba6c009f50 tx=0x7fba6c0047a0 comp rx=0 tx=0).stop 2024-03-26T02:01:37.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.368+0000 7fba80c61700 1 -- 172.21.15.173:0/3390387344 shutdown_connections 2024-03-26T02:01:37.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.368+0000 7fba80c61700 1 --2- 172.21.15.173:0/3390387344 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fba7c103c70 0x7fba7c0749b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:37.369 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.368+0000 7fba80c61700 1 --2- 172.21.15.173:0/3390387344 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fba7c1020d0 0x7fba7c075e20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:37.370 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.368+0000 7fba80c61700 1 --2- 172.21.15.173:0/3390387344 >> v2:172.21.15.72:6800/2923394687 conn(0x7fba5c06dac0 0x7fba5c06ff80 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:37.370 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.368+0000 7fba80c61700 1 --2- 172.21.15.173:0/3390387344 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fba7c1032d0 0x7fba7c074470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:37.370 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.368+0000 7fba80c61700 1 -- 172.21.15.173:0/3390387344 >> 172.21.15.173:0/3390387344 conn(0x7fba7c0fd690 msgr2=0x7fba7c106500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:37.370 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.368+0000 7fba80c61700 1 -- 172.21.15.173:0/3390387344 shutdown_connections 2024-03-26T02:01:37.370 INFO:teuthology.orchestra.run.smithi173.stderr:2024-03-26T02:01:37.368+0000 7fba80c61700 1 -- 172.21.15.173:0/3390387344 wait complete. 2024-03-26T02:01:38.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:38 smithi072 ceph-mon[41619]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 35 B/s, 2 objects/s recovering 2024-03-26T02:01:38.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:38 smithi072 ceph-mon[41619]: from='client.? 172.21.15.173:0/3390387344' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:01:38.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:38 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:01:38.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:38 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:01:38.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:38 smithi072 ceph-mon[30177]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 35 B/s, 2 objects/s recovering 2024-03-26T02:01:38.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:38 smithi072 ceph-mon[30177]: from='client.? 172.21.15.173:0/3390387344' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:01:38.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:38 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:01:38.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:38 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:01:38.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:38 smithi173 ceph-mon[33839]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 35 B/s, 2 objects/s recovering 2024-03-26T02:01:38.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:38 smithi173 ceph-mon[33839]: from='client.? 172.21.15.173:0/3390387344' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:01:38.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:38 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T02:01:38.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:38 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T02:01:38.896 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-03-26T02:01:38.897 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-26T02:01:38.897 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-26T02:01:38.939 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-26T02:01:38.940 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-26T02:01:38.940 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-26T02:01:39.185 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:01:40.455 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:01:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:01:40.455 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:40 smithi072 ceph-mon[30177]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 999 B/s rd, 0 op/s; 32 B/s, 2 objects/s recovering 2024-03-26T02:01:40.455 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:40 smithi173 ceph-mon[33839]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 999 B/s rd, 0 op/s; 32 B/s, 2 objects/s recovering 2024-03-26T02:01:40.456 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:40 smithi072 ceph-mon[41619]: pgmap v216: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 999 B/s rd, 0 op/s; 32 B/s, 2 objects/s recovering 2024-03-26T02:01:40.793 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:01:40 smithi173 podman[64748]: 2024-03-26 02:01:40.670986896 +0000 UTC m=+0.115765233 container create d514371d5abc718f0fe6b9935e92d85352658bbc41a519523a7f97d14596c6b1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:01:41.005 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.004+0000 7faa7ab26700 1 -- 172.21.15.72:0/646005089 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa7406a6d0 msgr2=0x7faa7410ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:41.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.004+0000 7faa7ab26700 1 --2- 172.21.15.72:0/646005089 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa7406a6d0 0x7faa7410ef10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7faa6c009fc0 tx=0x7faa6c00b330 comp rx=0 tx=0).stop 2024-03-26T02:01:41.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.004+0000 7faa7ab26700 1 -- 172.21.15.72:0/646005089 shutdown_connections 2024-03-26T02:01:41.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.004+0000 7faa7ab26700 1 --2- 172.21.15.72:0/646005089 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa7406a6d0 0x7faa7410ef10 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:41.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.004+0000 7faa7ab26700 1 --2- 172.21.15.72:0/646005089 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa741080d0 0x7faa741084b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:41.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.004+0000 7faa7ab26700 1 --2- 172.21.15.72:0/646005089 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa74069d30 0x7faa7406a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:41.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.004+0000 7faa7ab26700 1 -- 172.21.15.72:0/646005089 >> 172.21.15.72:0/646005089 conn(0x7faa74077eb0 msgr2=0x7faa740782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:41.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.004+0000 7faa7ab26700 1 -- 172.21.15.72:0/646005089 shutdown_connections 2024-03-26T02:01:41.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.004+0000 7faa7ab26700 1 -- 172.21.15.72:0/646005089 wait complete. 2024-03-26T02:01:41.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.004+0000 7faa7ab26700 1 Processor -- start 2024-03-26T02:01:41.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.004+0000 7faa7ab26700 1 -- start start 2024-03-26T02:01:41.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.005+0000 7faa7ab26700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa74069d30 0x7faa74197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:41.006 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.005+0000 7faa7ab26700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa7406a6d0 0x7faa74198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:41.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.005+0000 7faa7ab26700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa741080d0 0x7faa7419c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:41.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.005+0000 7faa7ab26700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7419ccc0 con 0x7faa741080d0 2024-03-26T02:01:41.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.005+0000 7faa7ab26700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7419ce30 con 0x7faa7406a6d0 2024-03-26T02:01:41.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.005+0000 7faa7ab26700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa7419cff0 con 0x7faa74069d30 2024-03-26T02:01:41.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.005+0000 7faa788c2700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa74069d30 0x7faa74197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:41.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.005+0000 7faa790c3700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa741080d0 0x7faa7419c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:41.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.005+0000 7faa790c3700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa741080d0 0x7faa7419c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:43062/0 (socket says 172.21.15.72:43062) 2024-03-26T02:01:41.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.005+0000 7faa788c2700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa74069d30 0x7faa74197de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:53086/0 (socket says 172.21.15.72:53086) 2024-03-26T02:01:41.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.005+0000 7faa790c3700 1 -- 172.21.15.72:0/910810271 learned_addr learned my addr 172.21.15.72:0/910810271 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:01:41.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.005+0000 7faa73fff700 1 --2- 172.21.15.72:0/910810271 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa7406a6d0 0x7faa74198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:41.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.005+0000 7faa790c3700 1 -- 172.21.15.72:0/910810271 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa74069d30 msgr2=0x7faa74197de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:41.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.005+0000 7faa790c3700 1 --2- 172.21.15.72:0/910810271 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa74069d30 0x7faa74197de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:41.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.005+0000 7faa790c3700 1 -- 172.21.15.72:0/910810271 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa7406a6d0 msgr2=0x7faa74198320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:41.007 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.005+0000 7faa790c3700 1 --2- 172.21.15.72:0/910810271 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa7406a6d0 0x7faa74198320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:41.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.005+0000 7faa790c3700 1 -- 172.21.15.72:0/910810271 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa600095f0 con 0x7faa741080d0 2024-03-26T02:01:41.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.005+0000 7faa788c2700 1 --2- 172.21.15.72:0/910810271 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa74069d30 0x7faa74197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:01:41.008 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.005+0000 7faa790c3700 1 --2- 172.21.15.72:0/910810271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa741080d0 0x7faa7419c6b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7faa6c009450 tx=0x7faa6c00b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:41.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.005+0000 7faa71ffb700 1 -- 172.21.15.72:0/910810271 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6c00e040 con 0x7faa741080d0 2024-03-26T02:01:41.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.006+0000 7faa7ab26700 1 -- 172.21.15.72:0/910810271 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa6c009c40 con 0x7faa741080d0 2024-03-26T02:01:41.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.006+0000 7faa71ffb700 1 -- 172.21.15.72:0/910810271 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7faa6c007700 con 0x7faa741080d0 2024-03-26T02:01:41.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.006+0000 7faa71ffb700 1 -- 172.21.15.72:0/910810271 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6c0052a0 con 0x7faa741080d0 2024-03-26T02:01:41.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.006+0000 7faa7ab26700 1 -- 172.21.15.72:0/910810271 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa7419d500 con 0x7faa741080d0 2024-03-26T02:01:41.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.007+0000 7faa7ab26700 1 -- 172.21.15.72:0/910810271 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa74103780 con 0x7faa741080d0 2024-03-26T02:01:41.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.007+0000 7faa71ffb700 1 -- 172.21.15.72:0/910810271 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 16) v1 ==== 91414+0+0 (secure 0 0 0) 0x7faa6c01d470 con 0x7faa741080d0 2024-03-26T02:01:41.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.008+0000 7faa71ffb700 1 --2- 172.21.15.72:0/910810271 >> v2:172.21.15.72:6800/2923394687 conn(0x7faa6406da20 0x7faa6406fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:41.009 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.008+0000 7faa788c2700 1 --2- 172.21.15.72:0/910810271 >> v2:172.21.15.72:6800/2923394687 conn(0x7faa6406da20 0x7faa6406fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:41.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.008+0000 7faa71ffb700 1 -- 172.21.15.72:0/910810271 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6001+0+0 (secure 0 0 0) 0x7faa6c08e900 con 0x7faa741080d0 2024-03-26T02:01:41.010 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.008+0000 7faa788c2700 1 --2- 172.21.15.72:0/910810271 >> v2:172.21.15.72:6800/2923394687 conn(0x7faa6406da20 0x7faa6406fee0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7faa68009de0 tx=0x7faa68009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:41.013 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.013+0000 7faa71ffb700 1 -- 172.21.15.72:0/910810271 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7faa6c0581f0 con 0x7faa741080d0 2024-03-26T02:01:41.266 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.266+0000 7faa7ab26700 1 -- 172.21.15.72:0/910810271 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7faa74199140 con 0x7faa741080d0 2024-03-26T02:01:41.269 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.269+0000 7faa71ffb700 1 -- 172.21.15.72:0/910810271 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v16) v1 ==== 74+0+174790 (secure 0 0 0) 0x7faa6c05b8b0 con 0x7faa741080d0 2024-03-26T02:01:41.272 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T02:01:41.273 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.272+0000 7faa7ab26700 1 -- 172.21.15.72:0/910810271 >> v2:172.21.15.72:6800/2923394687 conn(0x7faa6406da20 msgr2=0x7faa6406fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:41.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.272+0000 7faa7ab26700 1 --2- 172.21.15.72:0/910810271 >> v2:172.21.15.72:6800/2923394687 conn(0x7faa6406da20 0x7faa6406fee0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7faa68009de0 tx=0x7faa68009190 comp rx=0 tx=0).stop 2024-03-26T02:01:41.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.272+0000 7faa7ab26700 1 -- 172.21.15.72:0/910810271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa741080d0 msgr2=0x7faa7419c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:41.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.272+0000 7faa7ab26700 1 --2- 172.21.15.72:0/910810271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa741080d0 0x7faa7419c6b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7faa6c009450 tx=0x7faa6c00b9f0 comp rx=0 tx=0).stop 2024-03-26T02:01:41.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.272+0000 7faa7ab26700 1 -- 172.21.15.72:0/910810271 shutdown_connections 2024-03-26T02:01:41.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.272+0000 7faa7ab26700 1 --2- 172.21.15.72:0/910810271 >> v2:172.21.15.72:6800/2923394687 conn(0x7faa6406da20 0x7faa6406fee0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:41.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.272+0000 7faa7ab26700 1 --2- 172.21.15.72:0/910810271 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7faa741080d0 0x7faa7419c6b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:41.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.272+0000 7faa7ab26700 1 --2- 172.21.15.72:0/910810271 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7faa74069d30 0x7faa74197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:41.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.272+0000 7faa7ab26700 1 --2- 172.21.15.72:0/910810271 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faa7406a6d0 0x7faa74198320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:41.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.272+0000 7faa7ab26700 1 -- 172.21.15.72:0/910810271 >> 172.21.15.72:0/910810271 conn(0x7faa74077eb0 msgr2=0x7faa74105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:41.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.272+0000 7faa7ab26700 1 -- 172.21.15.72:0/910810271 shutdown_connections 2024-03-26T02:01:41.274 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:41.272+0000 7faa7ab26700 1 -- 172.21.15.72:0/910810271 wait complete. 2024-03-26T02:01:41.282 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:01:40 smithi173 podman[64748]: 2024-03-26 02:01:40.923798627 +0000 UTC m=+0.368576964 container init d514371d5abc718f0fe6b9935e92d85352658bbc41a519523a7f97d14596c6b1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:01:41.282 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:01:40 smithi173 podman[64748]: 2024-03-26 02:01:40.9288481 +0000 UTC m=+0.373626446 container start d514371d5abc718f0fe6b9935e92d85352658bbc41a519523a7f97d14596c6b1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:01:41.282 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:01:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:01:40.951Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-26T02:01:41.282 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:01:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:01:40.951Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-26T02:01:41.282 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:01:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:01:40.951Z caller=main.go:570 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi173 (none))" 2024-03-26T02:01:41.282 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:01:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:01:40.951Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T02:01:41.282 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:01:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:01:40.951Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T02:01:41.282 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:01:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:01:40.953Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T02:01:41.282 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:01:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:01:40.954Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-26T02:01:41.283 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:01:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:01:40.954Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-26T02:01:41.283 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:01:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:01:40.954Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-26T02:01:41.283 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:01:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:01:40.956Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T02:01:41.283 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:01:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:01:40.956Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.399µs 2024-03-26T02:01:41.283 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:01:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:01:40.956Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T02:01:41.283 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:01:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:01:40.956Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-26T02:01:41.283 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:01:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:01:40.956Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.633µs wal_replay_duration=752.676µs wbl_replay_duration=137ns total_replay_duration=790.506µs 2024-03-26T02:01:41.283 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:01:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:01:40.958Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T02:01:41.283 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:01:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:01:40.958Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-26T02:01:41.283 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:01:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:01:40.958Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T02:01:41.283 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:01:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:01:40.975Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.112217ms db_storage=790ns remote_storage=1.478µs web_handler=363ns query_engine=445ns scrape=520.576µs scrape_sd=88.593µs notify=878ns notify_sd=2.084µs rules=16.265855ms tracing=4.847µs 2024-03-26T02:01:41.283 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:01:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:01:40.975Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-26T02:01:41.283 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:01:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:01:40.975Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-26T02:01:41.284 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:01:41 smithi173 bash[64748]: d514371d5abc718f0fe6b9935e92d85352658bbc41a519523a7f97d14596c6b1 2024-03-26T02:01:41.284 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:01:41 smithi173 systemd[1]: Started Ceph prometheus.a for c86fbdd4-eb13-11ee-95d0-87774f69a715. 2024-03-26T02:01:42.060 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6800","nonce":2923394687}]},"active_addr":"172.21.15.72:6800/2923394687","active_change":"2024-03-26T01:56:27.069990+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24115,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.72:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":1042368575}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":3915923670}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":3847840810}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":2684590485}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.72:0","nonce":1271789647}]}]} 2024-03-26T02:01:42.064 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-26T02:01:42.064 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-26T02:01:42.064 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T02:01:42.324 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:01:42.446 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:42 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:01:42.446 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:42 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:42.446 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:42 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:42.446 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:42 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:42.446 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:42 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T02:01:42.446 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:42 smithi072 ceph-mon[30177]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-26T02:01:42.446 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:42 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/910810271' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T02:01:42.447 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:42 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ignoring --setuser ceph since I am not root 2024-03-26T02:01:42.447 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:42 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ignoring --setgroup ceph since I am not root 2024-03-26T02:01:42.447 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:42 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:01:42.447 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:42 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:42.447 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:42 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:42.447 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:42 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:42.447 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:42 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T02:01:42.447 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:42 smithi072 ceph-mon[41619]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-26T02:01:42.447 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:42 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/910810271' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T02:01:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:42 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:01:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:42 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:42 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:42 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' 2024-03-26T02:01:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:42 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T02:01:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:42 smithi173 ceph-mon[33839]: pgmap v217: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-26T02:01:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:42 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/910810271' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T02:01:42.564 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:42 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: ignoring --setuser ceph since I am not root 2024-03-26T02:01:42.564 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:42 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: ignoring --setgroup ceph since I am not root 2024-03-26T02:01:42.564 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:42 smithi173 ceph-mgr[35973]: -- 172.21.15.173:0/46249770 <== mon.1 v2:172.21.15.173:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558dac6da1a0 con 0x558dac6b0c00 2024-03-26T02:01:42.968 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:42 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:01:42.715+0000 7f5d36bd7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:01:42.969 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:42 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:01:42.817+0000 7f5d36bd7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:01:43.063 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:42 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T02:01:42.706+0000 7f071f26e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T02:01:43.063 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:42 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T02:01:42.800+0000 7f071f26e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T02:01:43.438 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:43 smithi173 ceph-mon[33839]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T02:01:43.438 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:43 smithi173 ceph-mon[33839]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-26T02:01:43.462 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:43 smithi072 ceph-mon[30177]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T02:01:43.462 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:43 smithi072 ceph-mon[30177]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-26T02:01:43.463 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:43 smithi072 ceph-mon[41619]: from='mgr.14152 172.21.15.72:0/4265597173' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T02:01:43.463 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:43 smithi072 ceph-mon[41619]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-26T02:01:43.718 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:43 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:01:43.461+0000 7f5d36bd7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:01:43.719 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:43 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:01:43.556+0000 7f5d36bd7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:01:43.719 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:43 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:01:43.702+0000 7f5d36bd7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:01:43.772 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:43 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T02:01:43.437+0000 7f071f26e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T02:01:43.772 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:43 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T02:01:43.531+0000 7f071f26e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T02:01:43.772 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:43 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T02:01:43.675+0000 7f071f26e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T02:01:43.772 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:43 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T02:01:43.771+0000 7f071f26e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:01:44.003 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:43 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:01:43.800+0000 7f5d36bd7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T02:01:44.003 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:43 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:01:43.887+0000 7f5d36bd7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:01:44.062 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:43 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T02:01:43.854+0000 7f071f26e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T02:01:44.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.104+0000 7fcba8d91700 1 -- 172.21.15.72:0/4278080052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcba4069d30 msgr2=0x7fcba406a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:44.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.104+0000 7fcba8d91700 1 --2- 172.21.15.72:0/4278080052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcba4069d30 0x7fcba406a190 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcb98009920 tx=0x7fcb98009c30 comp rx=0 tx=0).stop 2024-03-26T02:01:44.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.104+0000 7fcba8d91700 1 -- 172.21.15.72:0/4278080052 shutdown_connections 2024-03-26T02:01:44.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.104+0000 7fcba8d91700 1 --2- 172.21.15.72:0/4278080052 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcba406a6d0 0x7fcba410ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:44.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.104+0000 7fcba8d91700 1 --2- 172.21.15.72:0/4278080052 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcba41080d0 0x7fcba41084b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:44.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.104+0000 7fcba8d91700 1 --2- 172.21.15.72:0/4278080052 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcba4069d30 0x7fcba406a190 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:44.105 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.104+0000 7fcba8d91700 1 -- 172.21.15.72:0/4278080052 >> 172.21.15.72:0/4278080052 conn(0x7fcba4077eb0 msgr2=0x7fcba40782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:44.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.104+0000 7fcba8d91700 1 -- 172.21.15.72:0/4278080052 shutdown_connections 2024-03-26T02:01:44.106 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.104+0000 7fcba8d91700 1 -- 172.21.15.72:0/4278080052 wait complete. 2024-03-26T02:01:44.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.105+0000 7fcba8d91700 1 Processor -- start 2024-03-26T02:01:44.107 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.105+0000 7fcba8d91700 1 -- start start 2024-03-26T02:01:44.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.105+0000 7fcba8d91700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcba4069d30 0x7fcba4197d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:44.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.105+0000 7fcba8d91700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcba406a6d0 0x7fcba4198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:44.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.105+0000 7fcba8d91700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcba41080d0 0x7fcba419c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:44.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.105+0000 7fcba8d91700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba419cbf0 con 0x7fcba41080d0 2024-03-26T02:01:44.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.105+0000 7fcba8d91700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcba419cd60 con 0x7fcba4069d30 2024-03-26T02:01:44.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.105+0000 7fcba8d91700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcba419cf20 con 0x7fcba406a6d0 2024-03-26T02:01:44.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.105+0000 7fcba1d9b700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcba406a6d0 0x7fcba4198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:44.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.105+0000 7fcba1d9b700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcba406a6d0 0x7fcba4198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:60976/0 (socket says 172.21.15.72:60976) 2024-03-26T02:01:44.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.105+0000 7fcba1d9b700 1 -- 172.21.15.72:0/2244513874 learned_addr learned my addr 172.21.15.72:0/2244513874 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:01:44.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.105+0000 7fcba259c700 1 --2- 172.21.15.72:0/2244513874 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcba4069d30 0x7fcba4197d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:44.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.105+0000 7fcba2d9d700 1 --2- 172.21.15.72:0/2244513874 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcba41080d0 0x7fcba419c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:44.108 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.106+0000 7fcba1d9b700 1 -- 172.21.15.72:0/2244513874 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcba4069d30 msgr2=0x7fcba4197d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:44.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.106+0000 7fcba1d9b700 1 --2- 172.21.15.72:0/2244513874 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcba4069d30 0x7fcba4197d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:44.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.106+0000 7fcba1d9b700 1 -- 172.21.15.72:0/2244513874 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcba41080d0 msgr2=0x7fcba419c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:44.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.106+0000 7fcba1d9b700 1 --2- 172.21.15.72:0/2244513874 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcba41080d0 0x7fcba419c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:44.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.106+0000 7fcba1d9b700 1 -- 172.21.15.72:0/2244513874 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb98009520 con 0x7fcba406a6d0 2024-03-26T02:01:44.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.106+0000 7fcba2d9d700 1 --2- 172.21.15.72:0/2244513874 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcba41080d0 0x7fcba419c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:01:44.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.106+0000 7fcba1d9b700 1 --2- 172.21.15.72:0/2244513874 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcba406a6d0 0x7fcba4198250 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcb9800fa50 tx=0x7fcb9800fa80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:44.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.106+0000 7fcb937fe700 1 -- 172.21.15.72:0/2244513874 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9801c070 con 0x7fcba406a6d0 2024-03-26T02:01:44.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.106+0000 7fcba8d91700 1 -- 172.21.15.72:0/2244513874 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcba419d1a0 con 0x7fcba406a6d0 2024-03-26T02:01:44.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.106+0000 7fcb937fe700 1 -- 172.21.15.72:0/2244513874 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fcb9800ba90 con 0x7fcba406a6d0 2024-03-26T02:01:44.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.106+0000 7fcb937fe700 1 -- 172.21.15.72:0/2244513874 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb98004930 con 0x7fcba406a6d0 2024-03-26T02:01:44.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.106+0000 7fcba8d91700 1 -- 172.21.15.72:0/2244513874 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcba41a4a60 con 0x7fcba406a6d0 2024-03-26T02:01:44.109 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.107+0000 7fcba8d91700 1 -- 172.21.15.72:0/2244513874 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcba4068d40 con 0x7fcba406a6d0 2024-03-26T02:01:44.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.108+0000 7fcb937fe700 1 -- 172.21.15.72:0/2244513874 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 17) v1 ==== 91428+0+0 (secure 0 0 0) 0x7fcb98020070 con 0x7fcba406a6d0 2024-03-26T02:01:44.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.108+0000 7fcb937fe700 1 --2- 172.21.15.72:0/2244513874 >> v2:172.21.15.72:6800/2923394687 conn(0x7fcb8c06da70 0x7fcb8c06ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:44.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.108+0000 7fcba259c700 1 -- 172.21.15.72:0/2244513874 >> v2:172.21.15.72:6800/2923394687 conn(0x7fcb8c06da70 msgr2=0x7fcb8c06ff30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/2923394687 2024-03-26T02:01:44.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.108+0000 7fcb937fe700 1 -- 172.21.15.72:0/2244513874 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6001+0+0 (secure 0 0 0) 0x7fcb9808dff0 con 0x7fcba406a6d0 2024-03-26T02:01:44.110 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.108+0000 7fcba259c700 1 --2- 172.21.15.72:0/2244513874 >> v2:172.21.15.72:6800/2923394687 conn(0x7fcb8c06da70 0x7fcb8c06ff30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T02:01:44.113 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.113+0000 7fcb937fe700 1 -- 172.21.15.72:0/2244513874 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fcb980578e0 con 0x7fcba406a6d0 2024-03-26T02:01:44.276 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:01:44.161+0000 7f5d36bd7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T02:01:44.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.309+0000 7fcba259c700 1 -- 172.21.15.72:0/2244513874 >> v2:172.21.15.72:6800/2923394687 conn(0x7fcb8c06da70 msgr2=0x7fcb8c06ff30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/2923394687 2024-03-26T02:01:44.309 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.309+0000 7fcba259c700 1 --2- 172.21.15.72:0/2244513874 >> v2:172.21.15.72:6800/2923394687 conn(0x7fcb8c06da70 0x7fcb8c06ff30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T02:01:44.316 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.315+0000 7fcba8d91700 1 -- 172.21.15.72:0/2244513874 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcba41117c0 con 0x7fcba406a6d0 2024-03-26T02:01:44.317 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.317+0000 7fcb937fe700 1 -- 172.21.15.72:0/2244513874 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+20677 (secure 0 0 0) 0x7fcb9805afa0 con 0x7fcba406a6d0 2024-03-26T02:01:44.318 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T02:01:44.318 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":70,"fsid":"c86fbdd4-eb13-11ee-95d0-87774f69a715","created":"2024-03-26T01:55:31.587056+0000","modified":"2024-03-26T02:01:28.883690+0000","last_up_change":"2024-03-26T02:00:53.755156+0000","last_in_change":"2024-03-26T02:00:36.756317+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T01:58:47.401060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-26T02:01:06.444953+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-26T02:01:07.020416+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-26T02:01:08.521443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-26T02:01:10.462741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-26T02:01:12.536909+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f311af9-e18d-42fa-9032-8fa8081d2c4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6801","nonce":1438405769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1438405769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1438405769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6803","nonce":1438405769}]},"public_addr":"172.21.15.72:6801/1438405769","cluster_addr":"172.21.15.72:6802/1438405769","heartbeat_back_addr":"172.21.15.72:6804/1438405769","heartbeat_front_addr":"172.21.15.72:6803/1438405769","state":["exists","up"]},{"osd":1,"uuid":"9db9d82b-f680-418a-9192-88eb5fe854b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6805","nonce":2073231730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2073231730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":2073231730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6807","nonce":2073231730}]},"public_addr":"172.21.15.72:6805/2073231730","cluster_addr":"172.21.15.72:6806/2073231730","heartbeat_back_addr":"172.21.15.72:6808/2073231730","heartbeat_front_addr":"172.21.15.72:6807/2073231730","state":["exists","up"]},{"osd":2,"uuid":"65c5c06e-e84d-4b5a-8204-65399b224f1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6809","nonce":3459293761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3459293761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3459293761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6811","nonce":3459293761}]},"public_addr":"172.21.15.72:6809/3459293761","cluster_addr":"172.21.15.72:6810/3459293761","heartbeat_back_addr":"172.21.15.72:6812/3459293761","heartbeat_front_addr":"172.21.15.72:6811/3459293761","state":["exists","up"]},{"osd":3,"uuid":"4f2f428c-7a84-4a34-8f42-179cbf06778d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6813","nonce":2677886334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":2677886334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":2677886334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6815","nonce":2677886334}]},"public_addr":"172.21.15.72:6813/2677886334","cluster_addr":"172.21.15.72:6814/2677886334","heartbeat_back_addr":"172.21.15.72:6816/2677886334","heartbeat_front_addr":"172.21.15.72:6815/2677886334","state":["exists","up"]},{"osd":4,"uuid":"5b3e8749-94e2-4727-9d1e-075ef3e43efd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":510505310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6801","nonce":510505310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6803","nonce":510505310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":510505310}]},"public_addr":"172.21.15.173:6800/510505310","cluster_addr":"172.21.15.173:6801/510505310","heartbeat_back_addr":"172.21.15.173:6803/510505310","heartbeat_front_addr":"172.21.15.173:6802/510505310","state":["exists","up"]},{"osd":5,"uuid":"652ab183-9759-426c-83cd-c61a676b06c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1295375838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6805","nonce":1295375838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6807","nonce":1295375838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1295375838}]},"public_addr":"172.21.15.173:6804/1295375838","cluster_addr":"172.21.15.173:6805/1295375838","heartbeat_back_addr":"172.21.15.173:6807/1295375838","heartbeat_front_addr":"172.21.15.173:6806/1295375838","state":["exists","up"]},{"osd":6,"uuid":"7d77483d-4aec-4685-8048-ca58d3482de3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3732303478}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6809","nonce":3732303478}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6811","nonce":3732303478}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3732303478}]},"public_addr":"172.21.15.173:6808/3732303478","cluster_addr":"172.21.15.173:6809/3732303478","heartbeat_back_addr":"172.21.15.173:6811/3732303478","heartbeat_front_addr":"172.21.15.173:6810/3732303478","state":["exists","up"]},{"osd":7,"uuid":"a7dabfd5-7bcf-4f9b-947d-b84777af1ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2776185825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6813","nonce":2776185825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6815","nonce":2776185825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2776185825}]},"public_addr":"172.21.15.173:6812/2776185825","cluster_addr":"172.21.15.173:6813/2776185825","heartbeat_back_addr":"172.21.15.173:6815/2776185825","heartbeat_front_addr":"172.21.15.173:6814/2776185825","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:57:52.811815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:58:19.048706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:58:44.908816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:59:10.323471+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:59:34.072843+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:00:00.403810+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:00:26.925604+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:00:53.049477+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/2939358967":"2024-03-27T01:56:27.069870+0000","172.21.15.72:0/334665634":"2024-03-27T01:56:27.069870+0000","172.21.15.72:0/3270791191":"2024-03-27T01:56:00.482837+0000","172.21.15.72:0/116986624":"2024-03-27T01:56:27.069870+0000","172.21.15.72:6800/347688983":"2024-03-27T01:56:00.482837+0000","172.21.15.72:0/2616932429":"2024-03-27T01:56:00.482837+0000","172.21.15.72:6800/3912119602":"2024-03-27T01:56:27.069870+0000","172.21.15.72:0/3486073291":"2024-03-27T01:56:00.482837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T02:01:44.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.320+0000 7fcba8d91700 1 -- 172.21.15.72:0/2244513874 >> v2:172.21.15.72:6800/2923394687 conn(0x7fcb8c06da70 msgr2=0x7fcb8c06ff30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T02:01:44.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.320+0000 7fcba8d91700 1 --2- 172.21.15.72:0/2244513874 >> v2:172.21.15.72:6800/2923394687 conn(0x7fcb8c06da70 0x7fcb8c06ff30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:44.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.320+0000 7fcba8d91700 1 -- 172.21.15.72:0/2244513874 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcba406a6d0 msgr2=0x7fcba4198250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:44.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.320+0000 7fcba8d91700 1 --2- 172.21.15.72:0/2244513874 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcba406a6d0 0x7fcba4198250 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fcb9800fa50 tx=0x7fcb9800fa80 comp rx=0 tx=0).stop 2024-03-26T02:01:44.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.320+0000 7fcba8d91700 1 -- 172.21.15.72:0/2244513874 shutdown_connections 2024-03-26T02:01:44.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.320+0000 7fcba8d91700 1 --2- 172.21.15.72:0/2244513874 >> v2:172.21.15.72:6800/2923394687 conn(0x7fcb8c06da70 0x7fcb8c06ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:44.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.320+0000 7fcba8d91700 1 --2- 172.21.15.72:0/2244513874 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fcba41080d0 0x7fcba419c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:44.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.320+0000 7fcba8d91700 1 --2- 172.21.15.72:0/2244513874 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fcba4069d30 0x7fcba4197d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:44.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.320+0000 7fcba8d91700 1 --2- 172.21.15.72:0/2244513874 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fcba406a6d0 0x7fcba4198250 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:44.321 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.320+0000 7fcba8d91700 1 -- 172.21.15.72:0/2244513874 >> 172.21.15.72:0/2244513874 conn(0x7fcba4077eb0 msgr2=0x7fcba4108d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:44.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.321+0000 7fcba8d91700 1 -- 172.21.15.72:0/2244513874 shutdown_connections 2024-03-26T02:01:44.322 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:44.321+0000 7fcba8d91700 1 -- 172.21.15.72:0/2244513874 wait complete. 2024-03-26T02:01:44.369 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:44 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T02:01:44.117+0000 7f071f26e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T02:01:44.369 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:44 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T02:01:44.227+0000 7f071f26e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:01:44.369 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:44 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T02:01:44.317+0000 7f071f26e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:01:44.635 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:44 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2244513874' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:01:44.635 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:44 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2244513874' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:01:44.635 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:01:44.275+0000 7f5d36bd7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T02:01:44.635 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:01:44.371+0000 7f5d36bd7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T02:01:44.635 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:01:44.468+0000 7f5d36bd7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:01:44.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:44 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2244513874' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:01:44.813 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:44 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T02:01:44.412+0000 7f071f26e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T02:01:44.813 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:44 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T02:01:44.591+0000 7f071f26e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:01:44.968 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:01:44.647+0000 7f5d36bd7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T02:01:45.056 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-26T02:01:45.057 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T02:01:45.306 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:01:45.719 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:45 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:01:45.647+0000 7f5d36bd7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:01:45.813 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:45 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T02:01:45.548+0000 7f071f26e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T02:01:45.813 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:45 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T02:01:45.677+0000 7f071f26e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:01:46.115 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:45 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:01:45.780+0000 7f5d36bd7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T02:01:46.115 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:46 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:01:46.115+0000 7f5d36bd7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:01:46.301 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:46 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T02:01:46.013+0000 7f071f26e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T02:01:46.301 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:46 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T02:01:46.103+0000 7f071f26e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:01:46.301 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:46 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T02:01:46.300+0000 7f071f26e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:01:46.408 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:46 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:01:46.207+0000 7f5d36bd7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T02:01:46.408 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:46 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:01:46.407+0000 7f5d36bd7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:01:46.562 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:46 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T02:01:46.486+0000 7f071f26e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:01:46.660 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:46 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:01:46.602+0000 7f5d36bd7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T02:01:46.920 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:46 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:01:46.896+0000 7f5d36bd7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:01:47.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.029+0000 7f01ca78f700 1 -- 172.21.15.72:0/1911092305 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01c4106300 msgr2=0x7f01c410ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:47.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.029+0000 7f01ca78f700 1 --2- 172.21.15.72:0/1911092305 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01c4106300 0x7f01c410ded0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f01bc009f90 tx=0x7f01bc00b330 comp rx=0 tx=0).stop 2024-03-26T02:01:47.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.029+0000 7f01ca78f700 1 -- 172.21.15.72:0/1911092305 shutdown_connections 2024-03-26T02:01:47.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.029+0000 7f01ca78f700 1 --2- 172.21.15.72:0/1911092305 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01c4106300 0x7f01c410ded0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:47.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.029+0000 7f01ca78f700 1 --2- 172.21.15.72:0/1911092305 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f01c41019e0 0x7f01c4101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:47.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.029+0000 7f01ca78f700 1 --2- 172.21.15.72:0/1911092305 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01c4102390 0x7f01c4105c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:47.030 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.029+0000 7f01ca78f700 1 -- 172.21.15.72:0/1911092305 >> 172.21.15.72:0/1911092305 conn(0x7f01c4077580 msgr2=0x7f01c40fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:47.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.029+0000 7f01ca78f700 1 -- 172.21.15.72:0/1911092305 shutdown_connections 2024-03-26T02:01:47.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.029+0000 7f01ca78f700 1 -- 172.21.15.72:0/1911092305 wait complete. 2024-03-26T02:01:47.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.029+0000 7f01ca78f700 1 Processor -- start 2024-03-26T02:01:47.031 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.029+0000 7f01ca78f700 1 -- start start 2024-03-26T02:01:47.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.030+0000 7f01ca78f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01c41019e0 0x7f01c4197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:47.032 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.030+0000 7f01ca78f700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f01c4102390 0x7f01c4198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:47.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.030+0000 7f01ca78f700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01c4106300 0x7f01c419c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:47.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.030+0000 7f01ca78f700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c419ccc0 con 0x7f01c4106300 2024-03-26T02:01:47.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.030+0000 7f01ca78f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01c419ce30 con 0x7f01c41019e0 2024-03-26T02:01:47.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.030+0000 7f01ca78f700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01c419cff0 con 0x7f01c4102390 2024-03-26T02:01:47.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.030+0000 7f01c8d2c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01c4106300 0x7f01c419c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:47.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.030+0000 7f01c8d2c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01c4106300 0x7f01c419c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:51736/0 (socket says 172.21.15.72:51736) 2024-03-26T02:01:47.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.030+0000 7f01c8d2c700 1 -- 172.21.15.72:0/2856887217 learned_addr learned my addr 172.21.15.72:0/2856887217 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:01:47.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.030+0000 7f01c37fe700 1 --2- 172.21.15.72:0/2856887217 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f01c4102390 0x7f01c4198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:47.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.030+0000 7f01c3fff700 1 --2- 172.21.15.72:0/2856887217 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01c41019e0 0x7f01c4197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:47.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.030+0000 7f01c37fe700 1 -- 172.21.15.72:0/2856887217 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01c41019e0 msgr2=0x7f01c4197de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:47.033 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.030+0000 7f01c37fe700 1 --2- 172.21.15.72:0/2856887217 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01c41019e0 0x7f01c4197de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:47.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.030+0000 7f01c37fe700 1 -- 172.21.15.72:0/2856887217 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01c4106300 msgr2=0x7f01c419c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:47.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.030+0000 7f01c37fe700 1 --2- 172.21.15.72:0/2856887217 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01c4106300 0x7f01c419c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:47.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.030+0000 7f01c37fe700 1 -- 172.21.15.72:0/2856887217 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b80095f0 con 0x7f01c4102390 2024-03-26T02:01:47.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.030+0000 7f01c8d2c700 1 --2- 172.21.15.72:0/2856887217 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01c4106300 0x7f01c419c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:01:47.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.031+0000 7f01c37fe700 1 --2- 172.21.15.72:0/2856887217 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f01c4102390 0x7f01c4198320 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f01c40748a0 tx=0x7f01b400dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:47.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.031+0000 7f01c17fa700 1 -- 172.21.15.72:0/2856887217 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b400f890 con 0x7f01c4102390 2024-03-26T02:01:47.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.031+0000 7f01c17fa700 1 -- 172.21.15.72:0/2856887217 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f01b4010460 con 0x7f01c4102390 2024-03-26T02:01:47.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.031+0000 7f01ca78f700 1 -- 172.21.15.72:0/2856887217 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01bc009c40 con 0x7f01c4102390 2024-03-26T02:01:47.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.031+0000 7f01c17fa700 1 -- 172.21.15.72:0/2856887217 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01b400fea0 con 0x7f01c4102390 2024-03-26T02:01:47.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.031+0000 7f01ca78f700 1 -- 172.21.15.72:0/2856887217 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01c41a4960 con 0x7f01c4102390 2024-03-26T02:01:47.034 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.032+0000 7f01ca78f700 1 -- 172.21.15.72:0/2856887217 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01c4102790 con 0x7f01c4102390 2024-03-26T02:01:47.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.032+0000 7f01c17fa700 1 -- 172.21.15.72:0/2856887217 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 17) v1 ==== 91428+0+0 (secure 0 0 0) 0x7f01b400e610 con 0x7f01c4102390 2024-03-26T02:01:47.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.033+0000 7f01c17fa700 1 --2- 172.21.15.72:0/2856887217 >> v2:172.21.15.72:6800/2923394687 conn(0x7f01ac06da70 0x7f01ac06ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:47.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.033+0000 7f01c3fff700 1 -- 172.21.15.72:0/2856887217 >> v2:172.21.15.72:6800/2923394687 conn(0x7f01ac06da70 msgr2=0x7f01ac06ff30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/2923394687 2024-03-26T02:01:47.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.033+0000 7f01c17fa700 1 -- 172.21.15.72:0/2856887217 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6001+0+0 (secure 0 0 0) 0x7f01b408cf50 con 0x7f01c4102390 2024-03-26T02:01:47.035 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.033+0000 7f01c3fff700 1 --2- 172.21.15.72:0/2856887217 >> v2:172.21.15.72:6800/2923394687 conn(0x7f01ac06da70 0x7f01ac06ff30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-26T02:01:47.036 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:46 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T02:01:46.771+0000 7f071f26e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T02:01:47.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.042+0000 7f01c17fa700 1 -- 172.21.15.72:0/2856887217 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f01b40c1280 con 0x7f01c4102390 2024-03-26T02:01:47.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:47 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:01:47.167+0000 7f5d36bd7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:01:47.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.234+0000 7f01c3fff700 1 -- 172.21.15.72:0/2856887217 >> v2:172.21.15.72:6800/2923394687 conn(0x7f01ac06da70 msgr2=0x7f01ac06ff30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.72:6800/2923394687 2024-03-26T02:01:47.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.234+0000 7f01c3fff700 1 --2- 172.21.15.72:0/2856887217 >> v2:172.21.15.72:6800/2923394687 conn(0x7f01ac06da70 0x7f01ac06ff30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-26T02:01:47.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.243+0000 7f01ca78f700 1 -- 172.21.15.72:0/2856887217 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01c4199140 con 0x7f01c4102390 2024-03-26T02:01:47.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.246+0000 7f01c17fa700 1 -- 172.21.15.72:0/2856887217 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+20677 (secure 0 0 0) 0x7f01b4056ae0 con 0x7f01c4102390 2024-03-26T02:01:47.247 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T02:01:47.247 INFO:teuthology.orchestra.run.smithi072.stdout:{"epoch":70,"fsid":"c86fbdd4-eb13-11ee-95d0-87774f69a715","created":"2024-03-26T01:55:31.587056+0000","modified":"2024-03-26T02:01:28.883690+0000","last_up_change":"2024-03-26T02:00:53.755156+0000","last_in_change":"2024-03-26T02:00:36.756317+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T01:58:47.401060+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-26T02:01:06.444953+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-26T02:01:07.020416+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-26T02:01:08.521443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-26T02:01:10.462741+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-26T02:01:12.536909+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0f311af9-e18d-42fa-9032-8fa8081d2c4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6801","nonce":1438405769}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6802","nonce":1438405769}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6804","nonce":1438405769}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6803","nonce":1438405769}]},"public_addr":"172.21.15.72:6801/1438405769","cluster_addr":"172.21.15.72:6802/1438405769","heartbeat_back_addr":"172.21.15.72:6804/1438405769","heartbeat_front_addr":"172.21.15.72:6803/1438405769","state":["exists","up"]},{"osd":1,"uuid":"9db9d82b-f680-418a-9192-88eb5fe854b2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6805","nonce":2073231730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6806","nonce":2073231730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6808","nonce":2073231730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6807","nonce":2073231730}]},"public_addr":"172.21.15.72:6805/2073231730","cluster_addr":"172.21.15.72:6806/2073231730","heartbeat_back_addr":"172.21.15.72:6808/2073231730","heartbeat_front_addr":"172.21.15.72:6807/2073231730","state":["exists","up"]},{"osd":2,"uuid":"65c5c06e-e84d-4b5a-8204-65399b224f1b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6809","nonce":3459293761}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6810","nonce":3459293761}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6812","nonce":3459293761}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6811","nonce":3459293761}]},"public_addr":"172.21.15.72:6809/3459293761","cluster_addr":"172.21.15.72:6810/3459293761","heartbeat_back_addr":"172.21.15.72:6812/3459293761","heartbeat_front_addr":"172.21.15.72:6811/3459293761","state":["exists","up"]},{"osd":3,"uuid":"4f2f428c-7a84-4a34-8f42-179cbf06778d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6813","nonce":2677886334}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6814","nonce":2677886334}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6816","nonce":2677886334}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.72:6815","nonce":2677886334}]},"public_addr":"172.21.15.72:6813/2677886334","cluster_addr":"172.21.15.72:6814/2677886334","heartbeat_back_addr":"172.21.15.72:6816/2677886334","heartbeat_front_addr":"172.21.15.72:6815/2677886334","state":["exists","up"]},{"osd":4,"uuid":"5b3e8749-94e2-4727-9d1e-075ef3e43efd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":510505310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6801","nonce":510505310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6803","nonce":510505310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":510505310}]},"public_addr":"172.21.15.173:6800/510505310","cluster_addr":"172.21.15.173:6801/510505310","heartbeat_back_addr":"172.21.15.173:6803/510505310","heartbeat_front_addr":"172.21.15.173:6802/510505310","state":["exists","up"]},{"osd":5,"uuid":"652ab183-9759-426c-83cd-c61a676b06c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1295375838}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6805","nonce":1295375838}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6807","nonce":1295375838}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1295375838}]},"public_addr":"172.21.15.173:6804/1295375838","cluster_addr":"172.21.15.173:6805/1295375838","heartbeat_back_addr":"172.21.15.173:6807/1295375838","heartbeat_front_addr":"172.21.15.173:6806/1295375838","state":["exists","up"]},{"osd":6,"uuid":"7d77483d-4aec-4685-8048-ca58d3482de3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":3732303478}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6809","nonce":3732303478}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6811","nonce":3732303478}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":3732303478}]},"public_addr":"172.21.15.173:6808/3732303478","cluster_addr":"172.21.15.173:6809/3732303478","heartbeat_back_addr":"172.21.15.173:6811/3732303478","heartbeat_front_addr":"172.21.15.173:6810/3732303478","state":["exists","up"]},{"osd":7,"uuid":"a7dabfd5-7bcf-4f9b-947d-b84777af1ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2776185825}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6813","nonce":2776185825}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6815","nonce":2776185825}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2776185825}]},"public_addr":"172.21.15.173:6812/2776185825","cluster_addr":"172.21.15.173:6813/2776185825","heartbeat_back_addr":"172.21.15.173:6815/2776185825","heartbeat_front_addr":"172.21.15.173:6814/2776185825","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:57:52.811815+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:58:19.048706+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:58:44.908816+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:59:10.323471+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T01:59:34.072843+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:00:00.403810+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:00:26.925604+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T02:00:53.049477+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.9","mappings":[{"from":1,"to":2}]},{"pgid":"4.d","mappings":[{"from":1,"to":5}]},{"pgid":"4.14","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.72:0/2939358967":"2024-03-27T01:56:27.069870+0000","172.21.15.72:0/334665634":"2024-03-27T01:56:27.069870+0000","172.21.15.72:0/3270791191":"2024-03-27T01:56:00.482837+0000","172.21.15.72:0/116986624":"2024-03-27T01:56:27.069870+0000","172.21.15.72:6800/347688983":"2024-03-27T01:56:00.482837+0000","172.21.15.72:0/2616932429":"2024-03-27T01:56:00.482837+0000","172.21.15.72:6800/3912119602":"2024-03-27T01:56:27.069870+0000","172.21.15.72:0/3486073291":"2024-03-27T01:56:00.482837+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T02:01:47.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.248+0000 7f01ca78f700 1 -- 172.21.15.72:0/2856887217 >> v2:172.21.15.72:6800/2923394687 conn(0x7f01ac06da70 msgr2=0x7f01ac06ff30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T02:01:47.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.248+0000 7f01ca78f700 1 --2- 172.21.15.72:0/2856887217 >> v2:172.21.15.72:6800/2923394687 conn(0x7f01ac06da70 0x7f01ac06ff30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:47.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.248+0000 7f01ca78f700 1 -- 172.21.15.72:0/2856887217 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f01c4102390 msgr2=0x7f01c4198320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:47.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.248+0000 7f01ca78f700 1 --2- 172.21.15.72:0/2856887217 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f01c4102390 0x7f01c4198320 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f01c40748a0 tx=0x7f01b400dc30 comp rx=0 tx=0).stop 2024-03-26T02:01:47.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.248+0000 7f01ca78f700 1 -- 172.21.15.72:0/2856887217 shutdown_connections 2024-03-26T02:01:47.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.248+0000 7f01ca78f700 1 --2- 172.21.15.72:0/2856887217 >> v2:172.21.15.72:6800/2923394687 conn(0x7f01ac06da70 0x7f01ac06ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:47.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.248+0000 7f01ca78f700 1 --2- 172.21.15.72:0/2856887217 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f01c4106300 0x7f01c419c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:47.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.248+0000 7f01ca78f700 1 --2- 172.21.15.72:0/2856887217 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f01c41019e0 0x7f01c4197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:47.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.248+0000 7f01ca78f700 1 --2- 172.21.15.72:0/2856887217 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f01c4102390 0x7f01c4198320 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:47.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.248+0000 7f01ca78f700 1 -- 172.21.15.72:0/2856887217 >> 172.21.15.72:0/2856887217 conn(0x7f01c4077580 msgr2=0x7f01c4100870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:47.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.248+0000 7f01ca78f700 1 -- 172.21.15.72:0/2856887217 shutdown_connections 2024-03-26T02:01:47.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:47.248+0000 7f01ca78f700 1 -- 172.21.15.72:0/2856887217 wait complete. 2024-03-26T02:01:47.298 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:47 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T02:01:47.035+0000 7f071f26e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T02:01:47.298 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:47 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T02:01:47.271+0000 7f071f26e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:01:47.551 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:47 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2856887217' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:01:47.551 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:47 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:01:47.413+0000 7f5d36bd7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T02:01:47.551 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:47 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2856887217' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:01:47.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:47 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2856887217' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T02:01:47.856 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-26T02:01:47.857 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-26T02:01:47.857 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-26T02:01:47.857 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-26T02:01:47.858 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-26T02:01:47.858 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-26T02:01:47.859 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-26T02:01:47.859 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-26T02:01:48.362 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:01:48.362 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:01:48.364 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:01:48.365 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:01:48.389 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:01:48.393 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:01:48.404 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:01:48.562 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:48 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T02:01:48.075+0000 7f071f26e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:01:48.737 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:48 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:01:48.483+0000 7f5d36bd7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T02:01:48.966 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:48 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T02:01:48.651+0000 7f071f26e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:01:48.966 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:48 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T02:01:48.749+0000 7f071f26e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:01:48.966 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:48 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T02:01:48.965+0000 7f071f26e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:01:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:49 smithi173 ceph-mon[33839]: Standby manager daemon x restarted 2024-03-26T02:01:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:49 smithi173 ceph-mon[33839]: Standby manager daemon x started 2024-03-26T02:01:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:49 smithi173 ceph-mon[33839]: from='mgr.? 172.21.15.173:0/1254709360' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:01:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:49 smithi173 ceph-mon[33839]: from='mgr.? 172.21.15.173:0/1254709360' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:01:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:49 smithi173 ceph-mon[33839]: from='mgr.? 172.21.15.173:0/1254709360' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:01:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:49 smithi173 ceph-mon[33839]: from='mgr.? 172.21.15.173:0/1254709360' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:01:49.314 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:48 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: [26/Mar/2024:02:01:48] ENGINE Bus STARTING 2024-03-26T02:01:49.314 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:49 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: CherryPy Checker: 2024-03-26T02:01:49.314 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:49 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: The Application mounted at '' has an empty config. 2024-03-26T02:01:49.314 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:49 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: 2024-03-26T02:01:49.314 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:49 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: [26/Mar/2024:02:01:49] ENGINE Serving on http://:::9283 2024-03-26T02:01:49.314 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:01:49 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x[35951]: [26/Mar/2024:02:01:49] ENGINE Bus STARTED 2024-03-26T02:01:49.364 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:49 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:01:49.080+0000 7f5d36bd7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T02:01:49.365 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:49 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:01:49.179+0000 7f5d36bd7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T02:01:49.365 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:49 smithi072 ceph-mon[30177]: Standby manager daemon x restarted 2024-03-26T02:01:49.365 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:49 smithi072 ceph-mon[30177]: Standby manager daemon x started 2024-03-26T02:01:49.365 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:49 smithi072 ceph-mon[30177]: from='mgr.? 172.21.15.173:0/1254709360' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:01:49.365 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:49 smithi072 ceph-mon[30177]: from='mgr.? 172.21.15.173:0/1254709360' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:01:49.365 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:49 smithi072 ceph-mon[30177]: from='mgr.? 172.21.15.173:0/1254709360' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:01:49.365 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:49 smithi072 ceph-mon[30177]: from='mgr.? 172.21.15.173:0/1254709360' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:01:49.365 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:49 smithi072 ceph-mon[41619]: Standby manager daemon x restarted 2024-03-26T02:01:49.366 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:49 smithi072 ceph-mon[41619]: Standby manager daemon x started 2024-03-26T02:01:49.366 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:49 smithi072 ceph-mon[41619]: from='mgr.? 172.21.15.173:0/1254709360' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T02:01:49.366 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:49 smithi072 ceph-mon[41619]: from='mgr.? 172.21.15.173:0/1254709360' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T02:01:49.366 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:49 smithi072 ceph-mon[41619]: from='mgr.? 172.21.15.173:0/1254709360' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T02:01:49.366 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:49 smithi072 ceph-mon[41619]: from='mgr.? 172.21.15.173:0/1254709360' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T02:01:49.622 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:49 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:01:49.400+0000 7f5d36bd7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T02:01:49.622 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:49 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:01:49] ENGINE Bus STARTING 2024-03-26T02:01:49.951 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:49 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: CherryPy Checker: 2024-03-26T02:01:49.951 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:49 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: The Application mounted at '' has an empty config. 2024-03-26T02:01:49.951 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:49 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:01:49.951 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:49 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:01:49] ENGINE Serving on http://:::9283 2024-03-26T02:01:49.951 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:49 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:01:49] ENGINE Bus STARTED 2024-03-26T02:01:50.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T02:01:50.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: Active manager daemon y restarted 2024-03-26T02:01:50.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: Activating manager daemon y 2024-03-26T02:01:50.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T02:01:50.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: mgrmap e19: y(active, starting, since 0.00844532s), standbys: x 2024-03-26T02:01:50.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:01:50.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:01:50.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:01:50.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:01:50.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:01:50.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:01:50.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:01:50.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:01:50.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:01:50.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:01:50.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:01:50.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:01:50.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:01:50.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:01:50.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:01:50.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:01:50.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: Manager daemon y is now available 2024-03-26T02:01:50.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:01:50.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:01:50.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:01:50.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:01:50.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:01:50.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:01:50.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T02:01:50.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: Active manager daemon y restarted 2024-03-26T02:01:50.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: Activating manager daemon y 2024-03-26T02:01:50.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T02:01:50.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: mgrmap e19: y(active, starting, since 0.00844532s), standbys: x 2024-03-26T02:01:50.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:01:50.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:01:50.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:01:50.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:01:50.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:01:50.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:01:50.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:01:50.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:01:50.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:01:50.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:01:50.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:01:50.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:01:50.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:01:50.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:01:50.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:01:50.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:01:50.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: Manager daemon y is now available 2024-03-26T02:01:50.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:01:50.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:01:50.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:01:50.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:01:50.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:01:50.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:01:50.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T02:01:50.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: Active manager daemon y restarted 2024-03-26T02:01:50.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: Activating manager daemon y 2024-03-26T02:01:50.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T02:01:50.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: mgrmap e19: y(active, starting, since 0.00844532s), standbys: x 2024-03-26T02:01:50.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T02:01:50.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T02:01:50.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T02:01:50.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T02:01:50.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T02:01:50.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T02:01:50.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T02:01:50.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T02:01:50.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T02:01:50.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T02:01:50.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T02:01:50.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T02:01:50.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T02:01:50.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T02:01:50.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T02:01:50.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T02:01:50.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: Manager daemon y is now available 2024-03-26T02:01:50.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:01:50.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:01:50.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:01:50.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:01:50.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T02:01:50.316 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T02:01:50.760 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:01:50 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:01:51.534 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:51 smithi072 ceph-mon[30177]: mgrmap e20: y(active, since 1.01288s), standbys: x 2024-03-26T02:01:51.534 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:51 smithi072 ceph-mon[30177]: [26/Mar/2024:02:01:50] ENGINE Bus STARTING 2024-03-26T02:01:51.534 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:51 smithi072 ceph-mon[30177]: [26/Mar/2024:02:01:50] ENGINE Serving on http://172.21.15.72:8765 2024-03-26T02:01:51.534 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:51 smithi072 ceph-mon[30177]: [26/Mar/2024:02:01:50] ENGINE Serving on https://172.21.15.72:7150 2024-03-26T02:01:51.534 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:51 smithi072 ceph-mon[30177]: [26/Mar/2024:02:01:50] ENGINE Bus STARTED 2024-03-26T02:01:51.534 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:51 smithi072 ceph-mon[41619]: mgrmap e20: y(active, since 1.01288s), standbys: x 2024-03-26T02:01:51.534 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:51 smithi072 ceph-mon[41619]: [26/Mar/2024:02:01:50] ENGINE Bus STARTING 2024-03-26T02:01:51.534 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:51 smithi072 ceph-mon[41619]: [26/Mar/2024:02:01:50] ENGINE Serving on http://172.21.15.72:8765 2024-03-26T02:01:51.534 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:51 smithi072 ceph-mon[41619]: [26/Mar/2024:02:01:50] ENGINE Serving on https://172.21.15.72:7150 2024-03-26T02:01:51.535 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:51 smithi072 ceph-mon[41619]: [26/Mar/2024:02:01:50] ENGINE Bus STARTED 2024-03-26T02:01:51.709 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:51 smithi173 ceph-mon[33839]: mgrmap e20: y(active, since 1.01288s), standbys: x 2024-03-26T02:01:51.709 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:51 smithi173 ceph-mon[33839]: [26/Mar/2024:02:01:50] ENGINE Bus STARTING 2024-03-26T02:01:51.709 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:51 smithi173 ceph-mon[33839]: [26/Mar/2024:02:01:50] ENGINE Serving on http://172.21.15.72:8765 2024-03-26T02:01:51.709 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:51 smithi173 ceph-mon[33839]: [26/Mar/2024:02:01:50] ENGINE Serving on https://172.21.15.72:7150 2024-03-26T02:01:51.709 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:51 smithi173 ceph-mon[33839]: [26/Mar/2024:02:01:50] ENGINE Bus STARTED 2024-03-26T02:01:52.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:52 smithi072 ceph-mon[30177]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:01:52.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:52 smithi072 ceph-mon[41619]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:01:52.812 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:52 smithi173 ceph-mon[33839]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:01:53.700 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:53 smithi072 ceph-mon[30177]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-26T02:01:53.701 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:53 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:01:53.701 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:53 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:01:53.701 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:53 smithi072 ceph-mon[41619]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-26T02:01:53.701 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:53 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:01:53.701 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:53 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:01:53.812 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:53 smithi173 ceph-mon[33839]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-26T02:01:53.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:53 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:01:53.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:53 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:01:54.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.719+0000 7fe560bef700 1 -- 172.21.15.72:0/3483118841 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe55c1009c0 msgr2=0x7fe55c100e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:54.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.719+0000 7fe560bef700 1 --2- 172.21.15.72:0/3483118841 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe55c1009c0 0x7fe55c100e40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe550009920 tx=0x7fe550009c30 comp rx=0 tx=0).stop 2024-03-26T02:01:54.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.719+0000 7fe560bef700 1 -- 172.21.15.72:0/3483118841 shutdown_connections 2024-03-26T02:01:54.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.719+0000 7fe560bef700 1 --2- 172.21.15.72:0/3483118841 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe55c101380 0x7fe55c10d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:54.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.719+0000 7fe560bef700 1 --2- 172.21.15.72:0/3483118841 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe55c1067a0 0x7fe55c106b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:54.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.719+0000 7fe560bef700 1 --2- 172.21.15.72:0/3483118841 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe55c1009c0 0x7fe55c100e40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:54.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.719+0000 7fe560bef700 1 -- 172.21.15.72:0/3483118841 >> 172.21.15.72:0/3483118841 conn(0x7fe55c0762c0 msgr2=0x7fe55c0766d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:54.724 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.720+0000 7fe560bef700 1 -- 172.21.15.72:0/3483118841 shutdown_connections 2024-03-26T02:01:54.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.720+0000 7fe560bef700 1 -- 172.21.15.72:0/3483118841 wait complete. 2024-03-26T02:01:54.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.720+0000 7fe560bef700 1 Processor -- start 2024-03-26T02:01:54.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.720+0000 7fe560bef700 1 -- start start 2024-03-26T02:01:54.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.721+0000 7fe560bef700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe55c1009c0 0x7fe55c19d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:54.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.721+0000 7fe560bef700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe55c101380 0x7fe55c19de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:54.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.721+0000 7fe560bef700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe55c1067a0 0x7fe55c197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:54.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.721+0000 7fe560bef700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55c198140 con 0x7fe55c101380 2024-03-26T02:01:54.725 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.721+0000 7fe560bef700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55c198280 con 0x7fe55c1067a0 2024-03-26T02:01:54.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.721+0000 7fe560bef700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe55c198440 con 0x7fe55c1009c0 2024-03-26T02:01:54.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.721+0000 7fe55a59c700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe55c1009c0 0x7fe55c19d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:54.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.721+0000 7fe559d9b700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe55c101380 0x7fe55c19de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:54.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.721+0000 7fe55a59c700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe55c1009c0 0x7fe55c19d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:59866/0 (socket says 172.21.15.72:59866) 2024-03-26T02:01:54.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.721+0000 7fe55a59c700 1 -- 172.21.15.72:0/1735507073 learned_addr learned my addr 172.21.15.72:0/1735507073 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:01:54.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.721+0000 7fe55ad9d700 1 --2- 172.21.15.72:0/1735507073 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe55c1067a0 0x7fe55c197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:54.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.721+0000 7fe559d9b700 1 -- 172.21.15.72:0/1735507073 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe55c1009c0 msgr2=0x7fe55c19d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:54.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.721+0000 7fe559d9b700 1 --2- 172.21.15.72:0/1735507073 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe55c1009c0 0x7fe55c19d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:54.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.721+0000 7fe559d9b700 1 -- 172.21.15.72:0/1735507073 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe55c1067a0 msgr2=0x7fe55c197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:54.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.721+0000 7fe559d9b700 1 --2- 172.21.15.72:0/1735507073 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe55c1067a0 0x7fe55c197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:54.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.721+0000 7fe559d9b700 1 -- 172.21.15.72:0/1735507073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe554009d00 con 0x7fe55c101380 2024-03-26T02:01:54.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.721+0000 7fe55a59c700 1 --2- 172.21.15.72:0/1735507073 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe55c1009c0 0x7fe55c19d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:01:54.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.721+0000 7fe559d9b700 1 --2- 172.21.15.72:0/1735507073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe55c101380 0x7fe55c19de60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe550000c00 tx=0x7fe55000f9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:54.726 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.721+0000 7fe54b7fe700 1 -- 172.21.15.72:0/1735507073 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55001c070 con 0x7fe55c101380 2024-03-26T02:01:54.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.722+0000 7fe54b7fe700 1 -- 172.21.15.72:0/1735507073 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fe55000ba10 con 0x7fe55c101380 2024-03-26T02:01:54.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.722+0000 7fe560bef700 1 -- 172.21.15.72:0/1735507073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe550009520 con 0x7fe55c101380 2024-03-26T02:01:54.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.722+0000 7fe54b7fe700 1 -- 172.21.15.72:0/1735507073 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe550004930 con 0x7fe55c101380 2024-03-26T02:01:54.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.722+0000 7fe560bef700 1 -- 172.21.15.72:0/1735507073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe55c198940 con 0x7fe55c101380 2024-03-26T02:01:54.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.722+0000 7fe560bef700 1 -- 172.21.15.72:0/1735507073 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe55c10f940 con 0x7fe55c101380 2024-03-26T02:01:54.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.723+0000 7fe54b7fe700 1 -- 172.21.15.72:0/1735507073 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7fe550020070 con 0x7fe55c101380 2024-03-26T02:01:54.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.723+0000 7fe54b7fe700 1 --2- 172.21.15.72:0/1735507073 >> v2:172.21.15.72:6800/3246690770 conn(0x7fe54406da80 0x7fe54406ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:54.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.723+0000 7fe54b7fe700 1 -- 172.21.15.72:0/1735507073 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6259+0+0 (secure 0 0 0) 0x7fe55008e600 con 0x7fe55c101380 2024-03-26T02:01:54.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.723+0000 7fe55a59c700 1 --2- 172.21.15.72:0/1735507073 >> v2:172.21.15.72:6800/3246690770 conn(0x7fe54406da80 0x7fe54406ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:54.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.724+0000 7fe54b7fe700 1 --2- 172.21.15.72:0/1735507073 >> v2:172.21.15.173:6808/3732303478 conn(0x7fe544077320 0x7fe5440797c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:54.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.724+0000 7fe54b7fe700 1 -- 172.21.15.72:0/1735507073 --> v2:172.21.15.173:6808/3732303478 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe544079e50 con 0x7fe544077320 2024-03-26T02:01:54.727 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.724+0000 7fe54b7fe700 1 -- 172.21.15.72:0/1735507073 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe55008ea50 con 0x7fe55c101380 2024-03-26T02:01:54.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.724+0000 7fe55a59c700 1 --2- 172.21.15.72:0/1735507073 >> v2:172.21.15.72:6800/3246690770 conn(0x7fe54406da80 0x7fe54406ff40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe54c005950 tx=0x7fe54c00b600 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:54.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.724+0000 7fe55ad9d700 1 --2- 172.21.15.72:0/1735507073 >> v2:172.21.15.173:6808/3732303478 conn(0x7fe544077320 0x7fe5440797c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:54.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.724+0000 7fe55ad9d700 1 --2- 172.21.15.72:0/1735507073 >> v2:172.21.15.173:6808/3732303478 conn(0x7fe544077320 0x7fe5440797c0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:54.728 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.725+0000 7fe54b7fe700 1 -- 172.21.15.72:0/1735507073 <== osd.6 v2:172.21.15.173:6808/3732303478 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe544079e50 con 0x7fe544077320 2024-03-26T02:01:54.741 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.741+0000 7fe560bef700 1 -- 172.21.15.72:0/1735507073 --> v2:172.21.15.173:6808/3732303478 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe55c062710 con 0x7fe544077320 2024-03-26T02:01:54.742 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.741+0000 7fe54b7fe700 1 -- 172.21.15.72:0/1735507073 <== osd.6 v2:172.21.15.173:6808/3732303478 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe55c062710 con 0x7fe544077320 2024-03-26T02:01:54.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.742+0000 7fe560bef700 1 -- 172.21.15.72:0/1735507073 >> v2:172.21.15.173:6808/3732303478 conn(0x7fe544077320 msgr2=0x7fe5440797c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:54.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.742+0000 7fe560bef700 1 --2- 172.21.15.72:0/1735507073 >> v2:172.21.15.173:6808/3732303478 conn(0x7fe544077320 0x7fe5440797c0 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:54.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.742+0000 7fe560bef700 1 -- 172.21.15.72:0/1735507073 >> v2:172.21.15.72:6800/3246690770 conn(0x7fe54406da80 msgr2=0x7fe54406ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:54.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.742+0000 7fe560bef700 1 --2- 172.21.15.72:0/1735507073 >> v2:172.21.15.72:6800/3246690770 conn(0x7fe54406da80 0x7fe54406ff40 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fe54c005950 tx=0x7fe54c00b600 comp rx=0 tx=0).stop 2024-03-26T02:01:54.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.742+0000 7fe560bef700 1 -- 172.21.15.72:0/1735507073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe55c101380 msgr2=0x7fe55c19de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:54.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.742+0000 7fe560bef700 1 --2- 172.21.15.72:0/1735507073 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe55c101380 0x7fe55c19de60 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe550000c00 tx=0x7fe55000f9e0 comp rx=0 tx=0).stop 2024-03-26T02:01:54.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.742+0000 7fe55ad9d700 1 -- 172.21.15.72:0/1735507073 reap_dead start 2024-03-26T02:01:54.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.742+0000 7fe560bef700 1 -- 172.21.15.72:0/1735507073 shutdown_connections 2024-03-26T02:01:54.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.742+0000 7fe560bef700 1 -- 172.21.15.72:0/1735507073 >> 172.21.15.72:0/1735507073 conn(0x7fe55c0762c0 msgr2=0x7fe55c0fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:54.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.742+0000 7fe560bef700 1 -- 172.21.15.72:0/1735507073 shutdown_connections 2024-03-26T02:01:54.743 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.742+0000 7fe560bef700 1 -- 172.21.15.72:0/1735507073 wait complete. 2024-03-26T02:01:54.868 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:54 smithi072 ceph-mon[30177]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:01:54.868 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:54 smithi072 ceph-mon[30177]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-26T02:01:54.868 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:54 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:01:54.868 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:54 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:01:54.868 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:54 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:01:54.868 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:54 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:01:54.868 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:54 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:01:54.868 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:54 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:01:54.868 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:54 smithi072 ceph-mon[30177]: Adjusting osd_memory_target on smithi173 to 3760M 2024-03-26T02:01:54.869 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:54 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:01:54.869 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:54 smithi072 ceph-mon[41619]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:01:54.869 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:54 smithi072 ceph-mon[41619]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-26T02:01:54.869 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:54 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:01:54.869 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:54 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:01:54.869 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:54 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:01:54.869 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:54 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:01:54.869 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:54 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:01:54.869 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:54 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:01:54.869 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:54 smithi072 ceph-mon[41619]: Adjusting osd_memory_target on smithi173 to 3760M 2024-03-26T02:01:54.869 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:54 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:01:55.218 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:01:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:01:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:01:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:54 smithi173 ceph-mon[33839]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:01:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:54 smithi173 ceph-mon[33839]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-26T02:01:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:54 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:01:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:54 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:01:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:54 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:01:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:54 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:01:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:54 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:01:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:54 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:01:55.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:54 smithi173 ceph-mon[33839]: Adjusting osd_memory_target on smithi173 to 3760M 2024-03-26T02:01:55.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:54 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:01:56.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.865+0000 7f72ec4a8700 1 -- 172.21.15.72:0/2555948063 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f72e4102a90 msgr2=0x7f72e410eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:56.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.865+0000 7f72ec4a8700 1 --2- 172.21.15.72:0/2555948063 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f72e4102a90 0x7f72e410eed0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f72dc009f90 tx=0x7f72dc00b330 comp rx=0 tx=0).stop 2024-03-26T02:01:56.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.865+0000 7f72ec4a8700 1 -- 172.21.15.72:0/2555948063 shutdown_connections 2024-03-26T02:01:56.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.865+0000 7f72ec4a8700 1 --2- 172.21.15.72:0/2555948063 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f72e4102a90 0x7f72e410eed0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:56.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.865+0000 7f72ec4a8700 1 --2- 172.21.15.72:0/2555948063 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f72e41080b0 0x7f72e4108490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:56.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.865+0000 7f72ec4a8700 1 --2- 172.21.15.72:0/2555948063 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72e41020d0 0x7f72e4102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:56.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.865+0000 7f72ec4a8700 1 -- 172.21.15.72:0/2555948063 >> 172.21.15.72:0/2555948063 conn(0x7f72e40791b0 msgr2=0x7f72e40795c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:56.198 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.865+0000 7f72ec4a8700 1 -- 172.21.15.72:0/2555948063 shutdown_connections 2024-03-26T02:01:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.865+0000 7f72ec4a8700 1 -- 172.21.15.72:0/2555948063 wait complete. 2024-03-26T02:01:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.866+0000 7f72ec4a8700 1 Processor -- start 2024-03-26T02:01:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.866+0000 7f72ec4a8700 1 -- start start 2024-03-26T02:01:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.866+0000 7f72ec4a8700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f72e41020d0 0x7f72e4197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.866+0000 7f72ec4a8700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f72e4102a90 0x7f72e41982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.866+0000 7f72ec4a8700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72e41080b0 0x7f72e419c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.866+0000 7f72ec4a8700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e419cc80 con 0x7f72e4102a90 2024-03-26T02:01:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.866+0000 7f72ec4a8700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e419cdf0 con 0x7f72e41080b0 2024-03-26T02:01:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.866+0000 7f72ec4a8700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e419cfb0 con 0x7f72e41020d0 2024-03-26T02:01:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.866+0000 7f72ea244700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f72e41020d0 0x7f72e4197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.866+0000 7f72e9a43700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f72e4102a90 0x7f72e41982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.866+0000 7f72ea244700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f72e41020d0 0x7f72e4197da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:59880/0 (socket says 172.21.15.72:59880) 2024-03-26T02:01:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.866+0000 7f72ea244700 1 -- 172.21.15.72:0/693221149 learned_addr learned my addr 172.21.15.72:0/693221149 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:01:56.199 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.866+0000 7f72ea244700 1 -- 172.21.15.72:0/693221149 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72e41080b0 msgr2=0x7f72e419c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:56.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.866+0000 7f72eaa45700 1 --2- 172.21.15.72:0/693221149 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72e41080b0 0x7f72e419c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:56.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.867+0000 7f72ea244700 1 --2- 172.21.15.72:0/693221149 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72e41080b0 0x7f72e419c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:56.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.867+0000 7f72ea244700 1 -- 172.21.15.72:0/693221149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f72e4102a90 msgr2=0x7f72e41982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:56.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.867+0000 7f72ea244700 1 --2- 172.21.15.72:0/693221149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f72e4102a90 0x7f72e41982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:56.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.867+0000 7f72ea244700 1 -- 172.21.15.72:0/693221149 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e00095f0 con 0x7f72e41020d0 2024-03-26T02:01:56.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.867+0000 7f72e9a43700 1 --2- 172.21.15.72:0/693221149 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f72e4102a90 0x7f72e41982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:01:56.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.867+0000 7f72eaa45700 1 --2- 172.21.15.72:0/693221149 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f72e41080b0 0x7f72e419c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T02:01:56.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.867+0000 7f72ea244700 1 --2- 172.21.15.72:0/693221149 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f72e41020d0 0x7f72e4197da0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f72e000ec10 tx=0x7f72e000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:56.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.867+0000 7f72db7fe700 1 -- 172.21.15.72:0/693221149 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e000cd40 con 0x7f72e41020d0 2024-03-26T02:01:56.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.867+0000 7f72ec4a8700 1 -- 172.21.15.72:0/693221149 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72dc009c40 con 0x7f72e41020d0 2024-03-26T02:01:56.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.867+0000 7f72db7fe700 1 -- 172.21.15.72:0/693221149 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f72e00044d0 con 0x7f72e41020d0 2024-03-26T02:01:56.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.867+0000 7f72db7fe700 1 -- 172.21.15.72:0/693221149 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e0010430 con 0x7f72e41020d0 2024-03-26T02:01:56.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.867+0000 7f72ec4a8700 1 -- 172.21.15.72:0/693221149 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e41a4920 con 0x7f72e41020d0 2024-03-26T02:01:56.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.868+0000 7f72ec4a8700 1 -- 172.21.15.72:0/693221149 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f72e4111b10 con 0x7f72e41020d0 2024-03-26T02:01:56.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.868+0000 7f72db7fe700 1 -- 172.21.15.72:0/693221149 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7f72e0003fe0 con 0x7f72e41020d0 2024-03-26T02:01:56.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.869+0000 7f72db7fe700 1 --2- 172.21.15.72:0/693221149 >> v2:172.21.15.72:6800/3246690770 conn(0x7f72d006da80 0x7f72d006ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:56.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.869+0000 7f72e9a43700 1 --2- 172.21.15.72:0/693221149 >> v2:172.21.15.72:6800/3246690770 conn(0x7f72d006da80 0x7f72d006ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:56.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.869+0000 7f72db7fe700 1 -- 172.21.15.72:0/693221149 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6259+0+0 (secure 0 0 0) 0x7f72e0014070 con 0x7f72e41020d0 2024-03-26T02:01:56.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.869+0000 7f72db7fe700 1 --2- 172.21.15.72:0/693221149 >> v2:172.21.15.173:6812/2776185825 conn(0x7f72d0077320 0x7f72d00797c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:56.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.869+0000 7f72db7fe700 1 -- 172.21.15.72:0/693221149 --> v2:172.21.15.173:6812/2776185825 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f72d0079e50 con 0x7f72d0077320 2024-03-26T02:01:56.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.869+0000 7f72db7fe700 1 -- 172.21.15.72:0/693221149 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7f72e008df60 con 0x7f72e41020d0 2024-03-26T02:01:56.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.869+0000 7f72eaa45700 1 --2- 172.21.15.72:0/693221149 >> v2:172.21.15.173:6812/2776185825 conn(0x7f72d0077320 0x7f72d00797c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:56.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.869+0000 7f72e9a43700 1 --2- 172.21.15.72:0/693221149 >> v2:172.21.15.72:6800/3246690770 conn(0x7f72d006da80 0x7f72d006ff40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f72e4112360 tx=0x7f72d4009190 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:56.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.870+0000 7f72eaa45700 1 --2- 172.21.15.72:0/693221149 >> v2:172.21.15.173:6812/2776185825 conn(0x7f72d0077320 0x7f72d00797c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:56.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.871+0000 7f72db7fe700 1 -- 172.21.15.72:0/693221149 <== osd.7 v2:172.21.15.173:6812/2776185825 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f72d0079e50 con 0x7f72d0077320 2024-03-26T02:01:56.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.886+0000 7f72ec4a8700 1 -- 172.21.15.72:0/693221149 --> v2:172.21.15.173:6812/2776185825 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f72e41075d0 con 0x7f72d0077320 2024-03-26T02:01:56.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.887+0000 7f72db7fe700 1 -- 172.21.15.72:0/693221149 <== osd.7 v2:172.21.15.173:6812/2776185825 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f72e41075d0 con 0x7f72d0077320 2024-03-26T02:01:56.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.887+0000 7f72ec4a8700 1 -- 172.21.15.72:0/693221149 >> v2:172.21.15.173:6812/2776185825 conn(0x7f72d0077320 msgr2=0x7f72d00797c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:56.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.887+0000 7f72ec4a8700 1 --2- 172.21.15.72:0/693221149 >> v2:172.21.15.173:6812/2776185825 conn(0x7f72d0077320 0x7f72d00797c0 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:56.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.888+0000 7f72ec4a8700 1 -- 172.21.15.72:0/693221149 >> v2:172.21.15.72:6800/3246690770 conn(0x7f72d006da80 msgr2=0x7f72d006ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:56.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.888+0000 7f72ec4a8700 1 --2- 172.21.15.72:0/693221149 >> v2:172.21.15.72:6800/3246690770 conn(0x7f72d006da80 0x7f72d006ff40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f72e4112360 tx=0x7f72d4009190 comp rx=0 tx=0).stop 2024-03-26T02:01:56.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.888+0000 7f72ec4a8700 1 -- 172.21.15.72:0/693221149 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f72e41020d0 msgr2=0x7f72e4197da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:56.202 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.888+0000 7f72ec4a8700 1 --2- 172.21.15.72:0/693221149 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f72e41020d0 0x7f72e4197da0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f72e000ec10 tx=0x7f72e000ef20 comp rx=0 tx=0).stop 2024-03-26T02:01:56.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.888+0000 7f72eaa45700 1 -- 172.21.15.72:0/693221149 reap_dead start 2024-03-26T02:01:56.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.888+0000 7f72ec4a8700 1 -- 172.21.15.72:0/693221149 shutdown_connections 2024-03-26T02:01:56.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.888+0000 7f72ec4a8700 1 -- 172.21.15.72:0/693221149 >> 172.21.15.72:0/693221149 conn(0x7f72e40791b0 msgr2=0x7f72e4105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:56.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.888+0000 7f72ec4a8700 1 -- 172.21.15.72:0/693221149 shutdown_connections 2024-03-26T02:01:56.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:54.888+0000 7f72ec4a8700 1 -- 172.21.15.72:0/693221149 wait complete. 2024-03-26T02:01:56.790 INFO:teuthology.orchestra.run.smithi072.stdout:197568495633 2024-03-26T02:01:56.791 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-26T02:01:57.048 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:56 smithi072 ceph-mon[30177]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-26T02:01:57.049 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:56 smithi072 ceph-mon[41619]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-26T02:01:57.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:56 smithi173 ceph-mon[33839]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-26T02:01:57.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.377+0000 7f9181529700 1 -- 172.21.15.72:0/4271757426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f917c101260 msgr2=0x7f917c1016e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:57.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.377+0000 7f9181529700 1 --2- 172.21.15.72:0/4271757426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f917c101260 0x7f917c1016e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f91700099c0 tx=0x7f9170009cd0 comp rx=0 tx=0).stop 2024-03-26T02:01:57.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.377+0000 7f9181529700 1 -- 172.21.15.72:0/4271757426 shutdown_connections 2024-03-26T02:01:57.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.377+0000 7f9181529700 1 --2- 172.21.15.72:0/4271757426 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f917c101c20 0x7f917c10de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:57.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.377+0000 7f9181529700 1 --2- 172.21.15.72:0/4271757426 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f917c1070b0 0x7f917c107490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:57.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.377+0000 7f9181529700 1 --2- 172.21.15.72:0/4271757426 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f917c101260 0x7f917c1016e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:57.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.377+0000 7f9181529700 1 -- 172.21.15.72:0/4271757426 >> 172.21.15.72:0/4271757426 conn(0x7f917c077e40 msgr2=0x7f917c078250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:57.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.377+0000 7f9181529700 1 -- 172.21.15.72:0/4271757426 shutdown_connections 2024-03-26T02:01:57.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.377+0000 7f9181529700 1 -- 172.21.15.72:0/4271757426 wait complete. 2024-03-26T02:01:57.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.378+0000 7f9181529700 1 Processor -- start 2024-03-26T02:01:57.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.378+0000 7f9181529700 1 -- start start 2024-03-26T02:01:57.380 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.378+0000 7f9181529700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f917c101260 0x7f917c198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:57.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.378+0000 7f9181529700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f917c101c20 0x7f917c198fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:57.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.378+0000 7f9181529700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f917c1070b0 0x7f917c1a3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:57.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.378+0000 7f9181529700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917c199830 con 0x7f917c101c20 2024-03-26T02:01:57.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.378+0000 7f9181529700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917c1a4040 con 0x7f917c101260 2024-03-26T02:01:57.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.378+0000 7f9181529700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f917c1a4180 con 0x7f917c1070b0 2024-03-26T02:01:57.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.378+0000 7f917a7fc700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f917c101c20 0x7f917c198fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:57.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.378+0000 7f917a7fc700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f917c101c20 0x7f917c198fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47048/0 (socket says 172.21.15.72:47048) 2024-03-26T02:01:57.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.378+0000 7f917a7fc700 1 -- 172.21.15.72:0/473870388 learned_addr learned my addr 172.21.15.72:0/473870388 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:01:57.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.378+0000 7f917b7fe700 1 --2- 172.21.15.72:0/473870388 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f917c1070b0 0x7f917c1a3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:57.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.378+0000 7f917affd700 1 --2- 172.21.15.72:0/473870388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f917c101260 0x7f917c198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:57.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.379+0000 7f917a7fc700 1 -- 172.21.15.72:0/473870388 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f917c1070b0 msgr2=0x7f917c1a3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:57.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.379+0000 7f917a7fc700 1 --2- 172.21.15.72:0/473870388 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f917c1070b0 0x7f917c1a3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:57.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.379+0000 7f917a7fc700 1 -- 172.21.15.72:0/473870388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f917c101260 msgr2=0x7f917c198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:57.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.379+0000 7f917a7fc700 1 --2- 172.21.15.72:0/473870388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f917c101260 0x7f917c198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:57.381 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.379+0000 7f917a7fc700 1 -- 172.21.15.72:0/473870388 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916c009520 con 0x7f917c101c20 2024-03-26T02:01:57.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.379+0000 7f917affd700 1 --2- 172.21.15.72:0/473870388 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f917c101260 0x7f917c198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T02:01:57.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.379+0000 7f917a7fc700 1 --2- 172.21.15.72:0/473870388 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f917c101c20 0x7f917c198fc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9170009fd0 tx=0x7f917000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:57.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.379+0000 7f9163fff700 1 -- 172.21.15.72:0/473870388 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917001c070 con 0x7f917c101c20 2024-03-26T02:01:57.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.379+0000 7f9163fff700 1 -- 172.21.15.72:0/473870388 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f917000fe20 con 0x7f917c101c20 2024-03-26T02:01:57.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.379+0000 7f9181529700 1 -- 172.21.15.72:0/473870388 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91700095f0 con 0x7f917c101c20 2024-03-26T02:01:57.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.379+0000 7f9163fff700 1 -- 172.21.15.72:0/473870388 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9170022ce0 con 0x7f917c101c20 2024-03-26T02:01:57.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.379+0000 7f9181529700 1 -- 172.21.15.72:0/473870388 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f917c1a4630 con 0x7f917c101c20 2024-03-26T02:01:57.382 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.380+0000 7f9181529700 1 -- 172.21.15.72:0/473870388 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f917c1109e0 con 0x7f917c101c20 2024-03-26T02:01:57.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.380+0000 7f9163fff700 1 -- 172.21.15.72:0/473870388 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7f9170020070 con 0x7f917c101c20 2024-03-26T02:01:57.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.381+0000 7f9163fff700 1 --2- 172.21.15.72:0/473870388 >> v2:172.21.15.72:6800/3246690770 conn(0x7f916406da80 0x7f916406ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:57.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.381+0000 7f9163fff700 1 -- 172.21.15.72:0/473870388 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6259+0+0 (secure 0 0 0) 0x7f9170022e40 con 0x7f917c101c20 2024-03-26T02:01:57.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.381+0000 7f917affd700 1 --2- 172.21.15.72:0/473870388 >> v2:172.21.15.72:6800/3246690770 conn(0x7f916406da80 0x7f916406ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:57.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.381+0000 7f9163fff700 1 --2- 172.21.15.72:0/473870388 >> v2:172.21.15.72:6809/3459293761 conn(0x7f9164077320 0x7f91640797c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:57.383 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.381+0000 7f9163fff700 1 -- 172.21.15.72:0/473870388 --> v2:172.21.15.72:6809/3459293761 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9164079e50 con 0x7f9164077320 2024-03-26T02:01:57.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.381+0000 7f9163fff700 1 -- 172.21.15.72:0/473870388 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7f91700907e0 con 0x7f917c101c20 2024-03-26T02:01:57.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.381+0000 7f917b7fe700 1 --2- 172.21.15.72:0/473870388 >> v2:172.21.15.72:6809/3459293761 conn(0x7f9164077320 0x7f91640797c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:57.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.381+0000 7f917affd700 1 --2- 172.21.15.72:0/473870388 >> v2:172.21.15.72:6800/3246690770 conn(0x7f916406da80 0x7f916406ff40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f916c009e20 tx=0x7f916c009260 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:57.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.382+0000 7f917b7fe700 1 --2- 172.21.15.72:0/473870388 >> v2:172.21.15.72:6809/3459293761 conn(0x7f9164077320 0x7f91640797c0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:57.384 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.382+0000 7f9163fff700 1 -- 172.21.15.72:0/473870388 <== osd.2 v2:172.21.15.72:6809/3459293761 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9164079e50 con 0x7f9164077320 2024-03-26T02:01:57.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.399+0000 7f9181529700 1 -- 172.21.15.72:0/473870388 --> v2:172.21.15.72:6809/3459293761 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f917c1064c0 con 0x7f9164077320 2024-03-26T02:01:57.399 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.399+0000 7f9163fff700 1 -- 172.21.15.72:0/473870388 <== osd.2 v2:172.21.15.72:6809/3459293761 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f917c1064c0 con 0x7f9164077320 2024-03-26T02:01:57.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.399+0000 7f9181529700 1 -- 172.21.15.72:0/473870388 >> v2:172.21.15.72:6809/3459293761 conn(0x7f9164077320 msgr2=0x7f91640797c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:57.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.399+0000 7f9181529700 1 --2- 172.21.15.72:0/473870388 >> v2:172.21.15.72:6809/3459293761 conn(0x7f9164077320 0x7f91640797c0 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:57.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.399+0000 7f9181529700 1 -- 172.21.15.72:0/473870388 >> v2:172.21.15.72:6800/3246690770 conn(0x7f916406da80 msgr2=0x7f916406ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:57.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.399+0000 7f9181529700 1 --2- 172.21.15.72:0/473870388 >> v2:172.21.15.72:6800/3246690770 conn(0x7f916406da80 0x7f916406ff40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f916c009e20 tx=0x7f916c009260 comp rx=0 tx=0).stop 2024-03-26T02:01:57.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.399+0000 7f9181529700 1 -- 172.21.15.72:0/473870388 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f917c101c20 msgr2=0x7f917c198fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:57.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.399+0000 7f9181529700 1 --2- 172.21.15.72:0/473870388 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f917c101c20 0x7f917c198fc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9170009fd0 tx=0x7f917000f690 comp rx=0 tx=0).stop 2024-03-26T02:01:57.400 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.399+0000 7f917b7fe700 1 -- 172.21.15.72:0/473870388 reap_dead start 2024-03-26T02:01:57.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.399+0000 7f9181529700 1 -- 172.21.15.72:0/473870388 shutdown_connections 2024-03-26T02:01:57.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.399+0000 7f9181529700 1 -- 172.21.15.72:0/473870388 >> 172.21.15.72:0/473870388 conn(0x7f917c077e40 msgr2=0x7f917c104b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:57.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.400+0000 7f9181529700 1 -- 172.21.15.72:0/473870388 shutdown_connections 2024-03-26T02:01:57.401 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:57.400+0000 7f9181529700 1 -- 172.21.15.72:0/473870388 wait complete. 2024-03-26T02:01:58.025 INFO:teuthology.orchestra.run.smithi072.stdout:77309411368 2024-03-26T02:01:58.026 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-26T02:01:58.954 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:01:58 smithi072 ceph-mon[30177]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T02:01:58.955 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:01:58 smithi072 ceph-mon[41619]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T02:01:59.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:01:58 smithi173 ceph-mon[33839]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T02:01:59.599 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.598+0000 7f32b605c700 1 -- 172.21.15.72:0/2211068001 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32b0102ac0 msgr2=0x7f32b010ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:59.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.598+0000 7f32b605c700 1 --2- 172.21.15.72:0/2211068001 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32b0102ac0 0x7f32b010ef30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f32a800b0a0 tx=0x7f32a800b3b0 comp rx=0 tx=0).stop 2024-03-26T02:01:59.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.598+0000 7f32b605c700 1 -- 172.21.15.72:0/2211068001 shutdown_connections 2024-03-26T02:01:59.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.598+0000 7f32b605c700 1 --2- 172.21.15.72:0/2211068001 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32b0102ac0 0x7f32b010ef30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:59.657 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.598+0000 7f32b605c700 1 --2- 172.21.15.72:0/2211068001 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32b0108110 0x7f32b01084f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:59.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.598+0000 7f32b605c700 1 --2- 172.21.15.72:0/2211068001 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32b0102100 0x7f32b0102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:59.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.598+0000 7f32b605c700 1 -- 172.21.15.72:0/2211068001 >> 172.21.15.72:0/2211068001 conn(0x7f32b00791e0 msgr2=0x7f32b00795f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:59.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.598+0000 7f32b605c700 1 -- 172.21.15.72:0/2211068001 shutdown_connections 2024-03-26T02:01:59.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.598+0000 7f32b605c700 1 -- 172.21.15.72:0/2211068001 wait complete. 2024-03-26T02:01:59.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.598+0000 7f32b605c700 1 Processor -- start 2024-03-26T02:01:59.658 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.598+0000 7f32b605c700 1 -- start start 2024-03-26T02:01:59.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.599+0000 7f32b605c700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32b0102100 0x7f32b019d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:59.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.599+0000 7f32b605c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32b0102ac0 0x7f32b019de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:59.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.599+0000 7f32b605c700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32b0108110 0x7f32b0197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:59.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.599+0000 7f32b605c700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b0198130 con 0x7f32b0102ac0 2024-03-26T02:01:59.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.599+0000 7f32b605c700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b0198270 con 0x7f32b0108110 2024-03-26T02:01:59.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.599+0000 7f32b605c700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b0198430 con 0x7f32b0102100 2024-03-26T02:01:59.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.599+0000 7f32af7fe700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32b0102100 0x7f32b019d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:59.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.599+0000 7f32af7fe700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32b0102100 0x7f32b019d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:59914/0 (socket says 172.21.15.72:59914) 2024-03-26T02:01:59.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.599+0000 7f32af7fe700 1 -- 172.21.15.72:0/2200989711 learned_addr learned my addr 172.21.15.72:0/2200989711 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:01:59.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.599+0000 7f32aeffd700 1 --2- 172.21.15.72:0/2200989711 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32b0102ac0 0x7f32b019de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:59.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.599+0000 7f32affff700 1 --2- 172.21.15.72:0/2200989711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32b0108110 0x7f32b0197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:59.661 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.599+0000 7f32af7fe700 1 -- 172.21.15.72:0/2200989711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32b0108110 msgr2=0x7f32b0197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:59.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.599+0000 7f32af7fe700 1 --2- 172.21.15.72:0/2200989711 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32b0108110 0x7f32b0197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:59.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.599+0000 7f32af7fe700 1 -- 172.21.15.72:0/2200989711 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32b0102ac0 msgr2=0x7f32b019de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:59.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.599+0000 7f32af7fe700 1 --2- 172.21.15.72:0/2200989711 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32b0102ac0 0x7f32b019de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:59.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.599+0000 7f32af7fe700 1 -- 172.21.15.72:0/2200989711 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a4009520 con 0x7f32b0102100 2024-03-26T02:01:59.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.599+0000 7f32aeffd700 1 --2- 172.21.15.72:0/2200989711 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f32b0102ac0 0x7f32b019de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:01:59.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.599+0000 7f32af7fe700 1 --2- 172.21.15.72:0/2200989711 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32b0102100 0x7f32b019d910 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f32a000d950 tx=0x7f32a000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:59.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.599+0000 7f32acff9700 1 -- 172.21.15.72:0/2200989711 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a000f890 con 0x7f32b0102100 2024-03-26T02:01:59.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.599+0000 7f32b605c700 1 -- 172.21.15.72:0/2200989711 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a8009d00 con 0x7f32b0102100 2024-03-26T02:01:59.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.599+0000 7f32acff9700 1 -- 172.21.15.72:0/2200989711 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f32a0010460 con 0x7f32b0102100 2024-03-26T02:01:59.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.599+0000 7f32acff9700 1 -- 172.21.15.72:0/2200989711 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a000fea0 con 0x7f32b0102100 2024-03-26T02:01:59.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.599+0000 7f32b605c700 1 -- 172.21.15.72:0/2200989711 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b01a47e0 con 0x7f32b0102100 2024-03-26T02:01:59.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.600+0000 7f32b605c700 1 -- 172.21.15.72:0/2200989711 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f32b0111ad0 con 0x7f32b0102100 2024-03-26T02:01:59.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.601+0000 7f32acff9700 1 -- 172.21.15.72:0/2200989711 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7f32a000e610 con 0x7f32b0102100 2024-03-26T02:01:59.662 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.601+0000 7f32acff9700 1 --2- 172.21.15.72:0/2200989711 >> v2:172.21.15.72:6800/3246690770 conn(0x7f329806da80 0x7f329806ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:59.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.601+0000 7f32aeffd700 1 --2- 172.21.15.72:0/2200989711 >> v2:172.21.15.72:6800/3246690770 conn(0x7f329806da80 0x7f329806ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:59.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.601+0000 7f32acff9700 1 -- 172.21.15.72:0/2200989711 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6259+0+0 (secure 0 0 0) 0x7f32a008d1c0 con 0x7f32b0102100 2024-03-26T02:01:59.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.602+0000 7f32acff9700 1 --2- 172.21.15.72:0/2200989711 >> v2:172.21.15.173:6800/510505310 conn(0x7f32980772a0 0x7f3298079740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:01:59.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.602+0000 7f32acff9700 1 -- 172.21.15.72:0/2200989711 --> v2:172.21.15.173:6800/510505310 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3298079dd0 con 0x7f32980772a0 2024-03-26T02:01:59.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.602+0000 7f32acff9700 1 -- 172.21.15.72:0/2200989711 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32a008f880 con 0x7f32b0102100 2024-03-26T02:01:59.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.602+0000 7f32aeffd700 1 --2- 172.21.15.72:0/2200989711 >> v2:172.21.15.72:6800/3246690770 conn(0x7f329806da80 0x7f329806ff40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f32a4009fd0 tx=0x7f32a4009260 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:59.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.602+0000 7f32affff700 1 --2- 172.21.15.72:0/2200989711 >> v2:172.21.15.173:6800/510505310 conn(0x7f32980772a0 0x7f3298079740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:01:59.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.602+0000 7f32affff700 1 --2- 172.21.15.72:0/2200989711 >> v2:172.21.15.173:6800/510505310 conn(0x7f32980772a0 0x7f3298079740 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:01:59.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.603+0000 7f32acff9700 1 -- 172.21.15.72:0/2200989711 <== osd.4 v2:172.21.15.173:6800/510505310 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f3298079dd0 con 0x7f32980772a0 2024-03-26T02:01:59.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.619+0000 7f32b605c700 1 -- 172.21.15.72:0/2200989711 --> v2:172.21.15.173:6800/510505310 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f32b0107630 con 0x7f32980772a0 2024-03-26T02:01:59.663 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.620+0000 7f32acff9700 1 -- 172.21.15.72:0/2200989711 <== osd.4 v2:172.21.15.173:6800/510505310 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f32b0107630 con 0x7f32980772a0 2024-03-26T02:01:59.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.620+0000 7f32b605c700 1 -- 172.21.15.72:0/2200989711 >> v2:172.21.15.173:6800/510505310 conn(0x7f32980772a0 msgr2=0x7f3298079740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:59.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.620+0000 7f32b605c700 1 --2- 172.21.15.72:0/2200989711 >> v2:172.21.15.173:6800/510505310 conn(0x7f32980772a0 0x7f3298079740 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:01:59.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.620+0000 7f32b605c700 1 -- 172.21.15.72:0/2200989711 >> v2:172.21.15.72:6800/3246690770 conn(0x7f329806da80 msgr2=0x7f329806ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:59.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.620+0000 7f32b605c700 1 --2- 172.21.15.72:0/2200989711 >> v2:172.21.15.72:6800/3246690770 conn(0x7f329806da80 0x7f329806ff40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f32a4009fd0 tx=0x7f32a4009260 comp rx=0 tx=0).stop 2024-03-26T02:01:59.664 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.620+0000 7f32b605c700 1 -- 172.21.15.72:0/2200989711 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32b0102100 msgr2=0x7f32b019d910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:01:59.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.620+0000 7f32b605c700 1 --2- 172.21.15.72:0/2200989711 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f32b0102100 0x7f32b019d910 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f32a000d950 tx=0x7f32a000dc60 comp rx=0 tx=0).stop 2024-03-26T02:01:59.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.620+0000 7f32affff700 1 -- 172.21.15.72:0/2200989711 reap_dead start 2024-03-26T02:01:59.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.620+0000 7f32b605c700 1 -- 172.21.15.72:0/2200989711 shutdown_connections 2024-03-26T02:01:59.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.620+0000 7f32b605c700 1 -- 172.21.15.72:0/2200989711 >> 172.21.15.72:0/2200989711 conn(0x7f32b00791e0 msgr2=0x7f32b0105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:01:59.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.621+0000 7f32b605c700 1 -- 172.21.15.72:0/2200989711 shutdown_connections 2024-03-26T02:01:59.665 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:01:59.621+0000 7f32b605c700 1 -- 172.21.15.72:0/2200989711 wait complete. 2024-03-26T02:02:00.403 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.402+0000 7f089fe4d700 1 -- 172.21.15.72:0/1331697420 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f08981080b0 msgr2=0x7f0898108490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:00.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.402+0000 7f089fe4d700 1 --2- 172.21.15.72:0/1331697420 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f08981080b0 0x7f0898108490 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0894009990 tx=0x7f0894009ca0 comp rx=0 tx=0).stop 2024-03-26T02:02:00.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.403+0000 7f089fe4d700 1 -- 172.21.15.72:0/1331697420 shutdown_connections 2024-03-26T02:02:00.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.403+0000 7f089fe4d700 1 --2- 172.21.15.72:0/1331697420 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0898102a90 0x7f089810eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:00.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.403+0000 7f089fe4d700 1 --2- 172.21.15.72:0/1331697420 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f08981080b0 0x7f0898108490 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:00.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.403+0000 7f089fe4d700 1 --2- 172.21.15.72:0/1331697420 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08981020d0 0x7f0898102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:00.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.403+0000 7f089fe4d700 1 -- 172.21.15.72:0/1331697420 >> 172.21.15.72:0/1331697420 conn(0x7f08980791b0 msgr2=0x7f08980795c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:00.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.403+0000 7f089fe4d700 1 -- 172.21.15.72:0/1331697420 shutdown_connections 2024-03-26T02:02:00.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.403+0000 7f089fe4d700 1 -- 172.21.15.72:0/1331697420 wait complete. 2024-03-26T02:02:00.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.403+0000 7f089fe4d700 1 Processor -- start 2024-03-26T02:02:00.404 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.403+0000 7f089fe4d700 1 -- start start 2024-03-26T02:02:00.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.403+0000 7f089fe4d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08981020d0 0x7f089819d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:00.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.403+0000 7f089fe4d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0898102a90 0x7f089819dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:00.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.403+0000 7f089fe4d700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f08981080b0 0x7f0898197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:00.406 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.404+0000 7f089fe4d700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0898198080 con 0x7f0898102a90 2024-03-26T02:02:00.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.404+0000 7f089fe4d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08981981c0 con 0x7f08981020d0 2024-03-26T02:02:00.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.404+0000 7f089fe4d700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0898198380 con 0x7f08981080b0 2024-03-26T02:02:00.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.404+0000 7f089d3e8700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0898102a90 0x7f089819dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:00.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.404+0000 7f089e3ea700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f08981080b0 0x7f0898197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:00.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.404+0000 7f089e3ea700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f08981080b0 0x7f0898197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:59962/0 (socket says 172.21.15.72:59962) 2024-03-26T02:02:00.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.404+0000 7f089e3ea700 1 -- 172.21.15.72:0/1704534969 learned_addr learned my addr 172.21.15.72:0/1704534969 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:02:00.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.404+0000 7f089dbe9700 1 --2- 172.21.15.72:0/1704534969 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08981020d0 0x7f089819d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:00.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.404+0000 7f089e3ea700 1 -- 172.21.15.72:0/1704534969 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08981020d0 msgr2=0x7f089819d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:00.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.404+0000 7f089e3ea700 1 --2- 172.21.15.72:0/1704534969 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f08981020d0 0x7f089819d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:00.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.404+0000 7f089e3ea700 1 -- 172.21.15.72:0/1704534969 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0898102a90 msgr2=0x7f089819dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:00.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.404+0000 7f089e3ea700 1 --2- 172.21.15.72:0/1704534969 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0898102a90 0x7f089819dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:00.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.404+0000 7f089e3ea700 1 -- 172.21.15.72:0/1704534969 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0890009c40 con 0x7f08981080b0 2024-03-26T02:02:00.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.404+0000 7f089d3e8700 1 --2- 172.21.15.72:0/1704534969 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f0898102a90 0x7f089819dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:02:00.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.404+0000 7f089e3ea700 1 --2- 172.21.15.72:0/1704534969 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f08981080b0 0x7f0898197a20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f089000c370 tx=0x7f089000c680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:00.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.404+0000 7f088effd700 1 -- 172.21.15.72:0/1704534969 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0890017070 con 0x7f08981080b0 2024-03-26T02:02:00.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.404+0000 7f088effd700 1 -- 172.21.15.72:0/1704534969 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f089000f040 con 0x7f08981080b0 2024-03-26T02:02:00.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.404+0000 7f089fe4d700 1 -- 172.21.15.72:0/1704534969 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08940095f0 con 0x7f08981080b0 2024-03-26T02:02:00.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.404+0000 7f088effd700 1 -- 172.21.15.72:0/1704534969 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089000e050 con 0x7f08981080b0 2024-03-26T02:02:00.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.404+0000 7f089fe4d700 1 -- 172.21.15.72:0/1704534969 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08981a4730 con 0x7f08981080b0 2024-03-26T02:02:00.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.405+0000 7f089fe4d700 1 -- 172.21.15.72:0/1704534969 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0898111a30 con 0x7f08981080b0 2024-03-26T02:02:00.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.406+0000 7f088effd700 1 -- 172.21.15.72:0/1704534969 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7f089000cb60 con 0x7f08981080b0 2024-03-26T02:02:00.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.406+0000 7f088effd700 1 --2- 172.21.15.72:0/1704534969 >> v2:172.21.15.72:6800/3246690770 conn(0x7f088406da80 0x7f088406ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:00.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.406+0000 7f089dbe9700 1 --2- 172.21.15.72:0/1704534969 >> v2:172.21.15.72:6800/3246690770 conn(0x7f088406da80 0x7f088406ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:00.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.406+0000 7f088effd700 1 -- 172.21.15.72:0/1704534969 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6259+0+0 (secure 0 0 0) 0x7f089008cd60 con 0x7f08981080b0 2024-03-26T02:02:00.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.406+0000 7f088effd700 1 --2- 172.21.15.72:0/1704534969 >> v2:172.21.15.173:6804/1295375838 conn(0x7f0884077320 0x7f08840797c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:00.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.406+0000 7f088effd700 1 -- 172.21.15.72:0/1704534969 --> v2:172.21.15.173:6804/1295375838 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0884079e50 con 0x7f0884077320 2024-03-26T02:02:00.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.406+0000 7f088effd700 1 -- 172.21.15.72:0/1704534969 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7f089008f420 con 0x7f08981080b0 2024-03-26T02:02:00.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.407+0000 7f089dbe9700 1 --2- 172.21.15.72:0/1704534969 >> v2:172.21.15.72:6800/3246690770 conn(0x7f088406da80 0x7f088406ff40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0898198b60 tx=0x7f0894005960 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:00.409 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.407+0000 7f089d3e8700 1 --2- 172.21.15.72:0/1704534969 >> v2:172.21.15.173:6804/1295375838 conn(0x7f0884077320 0x7f08840797c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:00.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.407+0000 7f089d3e8700 1 --2- 172.21.15.72:0/1704534969 >> v2:172.21.15.173:6804/1295375838 conn(0x7f0884077320 0x7f08840797c0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:00.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.408+0000 7f088effd700 1 -- 172.21.15.72:0/1704534969 <== osd.5 v2:172.21.15.173:6804/1295375838 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0884079e50 con 0x7f0884077320 2024-03-26T02:02:00.424 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.424+0000 7f089fe4d700 1 -- 172.21.15.72:0/1704534969 --> v2:172.21.15.173:6804/1295375838 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f08981075d0 con 0x7f0884077320 2024-03-26T02:02:00.425 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.425+0000 7f088effd700 1 -- 172.21.15.72:0/1704534969 <== osd.5 v2:172.21.15.173:6804/1295375838 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f08981075d0 con 0x7f0884077320 2024-03-26T02:02:00.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.425+0000 7f089fe4d700 1 -- 172.21.15.72:0/1704534969 >> v2:172.21.15.173:6804/1295375838 conn(0x7f0884077320 msgr2=0x7f08840797c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:00.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.425+0000 7f089fe4d700 1 --2- 172.21.15.72:0/1704534969 >> v2:172.21.15.173:6804/1295375838 conn(0x7f0884077320 0x7f08840797c0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:00.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.425+0000 7f089fe4d700 1 -- 172.21.15.72:0/1704534969 >> v2:172.21.15.72:6800/3246690770 conn(0x7f088406da80 msgr2=0x7f088406ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:00.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.425+0000 7f089fe4d700 1 --2- 172.21.15.72:0/1704534969 >> v2:172.21.15.72:6800/3246690770 conn(0x7f088406da80 0x7f088406ff40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0898198b60 tx=0x7f0894005960 comp rx=0 tx=0).stop 2024-03-26T02:02:00.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.425+0000 7f089fe4d700 1 -- 172.21.15.72:0/1704534969 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f08981080b0 msgr2=0x7f0898197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:00.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.425+0000 7f089fe4d700 1 --2- 172.21.15.72:0/1704534969 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f08981080b0 0x7f0898197a20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f089000c370 tx=0x7f089000c680 comp rx=0 tx=0).stop 2024-03-26T02:02:00.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.425+0000 7f089e3ea700 1 -- 172.21.15.72:0/1704534969 reap_dead start 2024-03-26T02:02:00.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.425+0000 7f089fe4d700 1 -- 172.21.15.72:0/1704534969 shutdown_connections 2024-03-26T02:02:00.426 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.425+0000 7f089fe4d700 1 -- 172.21.15.72:0/1704534969 >> 172.21.15.72:0/1704534969 conn(0x7f08980791b0 msgr2=0x7f0898105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:00.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.425+0000 7f089fe4d700 1 -- 172.21.15.72:0/1704534969 shutdown_connections 2024-03-26T02:02:00.427 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.425+0000 7f089fe4d700 1 -- 172.21.15.72:0/1704534969 wait complete. 2024-03-26T02:02:00.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:02:00 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:02:00.826 INFO:teuthology.orchestra.run.smithi072.stdout:128849018910 2024-03-26T02:02:00.826 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-26T02:02:01.041 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.025+0000 7fdaa4f5d700 1 -- 172.21.15.72:0/95136209 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdaa0108180 msgr2=0x7fdaa0108560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:01.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.025+0000 7fdaa4f5d700 1 --2- 172.21.15.72:0/95136209 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdaa0108180 0x7fdaa0108560 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fda94009990 tx=0x7fda94009ca0 comp rx=0 tx=0).stop 2024-03-26T02:02:01.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.025+0000 7fdaa4f5d700 1 -- 172.21.15.72:0/95136209 shutdown_connections 2024-03-26T02:02:01.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.025+0000 7fdaa4f5d700 1 --2- 172.21.15.72:0/95136209 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdaa0102a90 0x7fdaa010ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:01.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.025+0000 7fdaa4f5d700 1 --2- 172.21.15.72:0/95136209 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdaa0108180 0x7fdaa0108560 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:01.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.025+0000 7fdaa4f5d700 1 --2- 172.21.15.72:0/95136209 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdaa01020d0 0x7fdaa0102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:01.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.025+0000 7fdaa4f5d700 1 -- 172.21.15.72:0/95136209 >> 172.21.15.72:0/95136209 conn(0x7fdaa00791b0 msgr2=0x7fdaa00795c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:01.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.025+0000 7fdaa4f5d700 1 -- 172.21.15.72:0/95136209 shutdown_connections 2024-03-26T02:02:01.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.025+0000 7fdaa4f5d700 1 -- 172.21.15.72:0/95136209 wait complete. 2024-03-26T02:02:01.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.026+0000 7fdaa4f5d700 1 Processor -- start 2024-03-26T02:02:01.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.026+0000 7fdaa4f5d700 1 -- start start 2024-03-26T02:02:01.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.026+0000 7fdaa4f5d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdaa01020d0 0x7fdaa019d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:01.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.026+0000 7fdaa4f5d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdaa0102a90 0x7fdaa019de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:01.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.026+0000 7fdaa4f5d700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdaa0108180 0x7fdaa0197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:01.042 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.026+0000 7fdaa4f5d700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa0198120 con 0x7fdaa0102a90 2024-03-26T02:02:01.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.026+0000 7fdaa4f5d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa0198260 con 0x7fdaa01020d0 2024-03-26T02:02:01.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.026+0000 7fdaa4f5d700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa0198420 con 0x7fdaa0108180 2024-03-26T02:02:01.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.026+0000 7fda9ed9d700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdaa0108180 0x7fdaa0197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:01.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.026+0000 7fda9dd9b700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdaa0102a90 0x7fdaa019de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:01.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.026+0000 7fda9dd9b700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdaa0102a90 0x7fdaa019de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:47132/0 (socket says 172.21.15.72:47132) 2024-03-26T02:02:01.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.026+0000 7fda9dd9b700 1 -- 172.21.15.72:0/162848864 learned_addr learned my addr 172.21.15.72:0/162848864 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:02:01.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.027+0000 7fda9e59c700 1 --2- 172.21.15.72:0/162848864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdaa01020d0 0x7fdaa019d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:01.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.027+0000 7fda9ed9d700 1 -- 172.21.15.72:0/162848864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdaa01020d0 msgr2=0x7fdaa019d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:01.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.027+0000 7fda9ed9d700 1 --2- 172.21.15.72:0/162848864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdaa01020d0 0x7fdaa019d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:01.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.027+0000 7fda9ed9d700 1 -- 172.21.15.72:0/162848864 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdaa0102a90 msgr2=0x7fdaa019de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:01.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.027+0000 7fda9ed9d700 1 --2- 172.21.15.72:0/162848864 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdaa0102a90 0x7fdaa019de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:01.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.027+0000 7fda9ed9d700 1 -- 172.21.15.72:0/162848864 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda98009c40 con 0x7fdaa0108180 2024-03-26T02:02:01.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.027+0000 7fda9e59c700 1 --2- 172.21.15.72:0/162848864 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdaa01020d0 0x7fdaa019d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T02:02:01.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.027+0000 7fda9dd9b700 1 --2- 172.21.15.72:0/162848864 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fdaa0102a90 0x7fdaa019de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:02:01.043 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.027+0000 7fda9ed9d700 1 --2- 172.21.15.72:0/162848864 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdaa0108180 0x7fdaa0197ac0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fda9800efb0 tx=0x7fda9800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:01.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.027+0000 7fda8f7fe700 1 -- 172.21.15.72:0/162848864 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9800ca50 con 0x7fdaa0108180 2024-03-26T02:02:01.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.027+0000 7fda8f7fe700 1 -- 172.21.15.72:0/162848864 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fda9800f040 con 0x7fdaa0108180 2024-03-26T02:02:01.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.027+0000 7fdaa4f5d700 1 -- 172.21.15.72:0/162848864 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda940095f0 con 0x7fdaa0108180 2024-03-26T02:02:01.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.027+0000 7fda8f7fe700 1 -- 172.21.15.72:0/162848864 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda980046a0 con 0x7fdaa0108180 2024-03-26T02:02:01.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.027+0000 7fdaa4f5d700 1 -- 172.21.15.72:0/162848864 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa01a47d0 con 0x7fdaa0108180 2024-03-26T02:02:01.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.028+0000 7fdaa4f5d700 1 -- 172.21.15.72:0/162848864 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdaa0111a90 con 0x7fdaa0108180 2024-03-26T02:02:01.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.028+0000 7fda8f7fe700 1 -- 172.21.15.72:0/162848864 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7fda98003a00 con 0x7fdaa0108180 2024-03-26T02:02:01.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.029+0000 7fda8f7fe700 1 --2- 172.21.15.72:0/162848864 >> v2:172.21.15.72:6800/3246690770 conn(0x7fda8806da80 0x7fda8806ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:01.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.029+0000 7fda8f7fe700 1 -- 172.21.15.72:0/162848864 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6259+0+0 (secure 0 0 0) 0x7fda9808d280 con 0x7fdaa0108180 2024-03-26T02:02:01.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.029+0000 7fda9e59c700 1 --2- 172.21.15.72:0/162848864 >> v2:172.21.15.72:6800/3246690770 conn(0x7fda8806da80 0x7fda8806ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:01.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.029+0000 7fda8f7fe700 1 --2- 172.21.15.72:0/162848864 >> v2:172.21.15.72:6813/2677886334 conn(0x7fda880773a0 0x7fda88079840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:01.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.029+0000 7fda8f7fe700 1 -- 172.21.15.72:0/162848864 --> v2:172.21.15.72:6813/2677886334 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda88079ed0 con 0x7fda880773a0 2024-03-26T02:02:01.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.029+0000 7fda8f7fe700 1 -- 172.21.15.72:0/162848864 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda9808f940 con 0x7fdaa0108180 2024-03-26T02:02:01.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.029+0000 7fda9dd9b700 1 --2- 172.21.15.72:0/162848864 >> v2:172.21.15.72:6813/2677886334 conn(0x7fda880773a0 0x7fda88079840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:01.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.029+0000 7fda9e59c700 1 --2- 172.21.15.72:0/162848864 >> v2:172.21.15.72:6800/3246690770 conn(0x7fda8806da80 0x7fda8806ff40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fda9400b7d0 tx=0x7fda94005fd0 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:01.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.029+0000 7fda9dd9b700 1 --2- 172.21.15.72:0/162848864 >> v2:172.21.15.72:6813/2677886334 conn(0x7fda880773a0 0x7fda88079840 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:01.044 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.030+0000 7fda8f7fe700 1 -- 172.21.15.72:0/162848864 <== osd.3 v2:172.21.15.72:6813/2677886334 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fda88079ed0 con 0x7fda880773a0 2024-03-26T02:02:01.048 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:02:01.048 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.048+0000 7fdaa4f5d700 1 -- 172.21.15.72:0/162848864 --> v2:172.21.15.72:6813/2677886334 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdaa0107630 con 0x7fda880773a0 2024-03-26T02:02:01.049 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:00 smithi072 ceph-mon[30177]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T02:02:01.049 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:00 smithi072 ceph-mon[41619]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T02:02:01.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.049+0000 7fda8f7fe700 1 -- 172.21.15.72:0/162848864 <== osd.3 v2:172.21.15.72:6813/2677886334 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fdaa0107630 con 0x7fda880773a0 2024-03-26T02:02:01.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.050+0000 7fdaa4f5d700 1 -- 172.21.15.72:0/162848864 >> v2:172.21.15.72:6813/2677886334 conn(0x7fda880773a0 msgr2=0x7fda88079840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:01.050 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.050+0000 7fdaa4f5d700 1 --2- 172.21.15.72:0/162848864 >> v2:172.21.15.72:6813/2677886334 conn(0x7fda880773a0 0x7fda88079840 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:01.052 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.051+0000 7fdaa4f5d700 1 -- 172.21.15.72:0/162848864 >> v2:172.21.15.72:6800/3246690770 conn(0x7fda8806da80 msgr2=0x7fda8806ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:01.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.051+0000 7fdaa4f5d700 1 --2- 172.21.15.72:0/162848864 >> v2:172.21.15.72:6800/3246690770 conn(0x7fda8806da80 0x7fda8806ff40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fda9400b7d0 tx=0x7fda94005fd0 comp rx=0 tx=0).stop 2024-03-26T02:02:01.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.051+0000 7fdaa4f5d700 1 -- 172.21.15.72:0/162848864 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdaa0108180 msgr2=0x7fdaa0197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:01.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.051+0000 7fdaa4f5d700 1 --2- 172.21.15.72:0/162848864 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fdaa0108180 0x7fdaa0197ac0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fda9800efb0 tx=0x7fda9800c3a0 comp rx=0 tx=0).stop 2024-03-26T02:02:01.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.051+0000 7fda9ed9d700 1 -- 172.21.15.72:0/162848864 reap_dead start 2024-03-26T02:02:01.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.051+0000 7fdaa4f5d700 1 -- 172.21.15.72:0/162848864 shutdown_connections 2024-03-26T02:02:01.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.051+0000 7fdaa4f5d700 1 -- 172.21.15.72:0/162848864 >> 172.21.15.72:0/162848864 conn(0x7fdaa00791b0 msgr2=0x7fdaa0105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:01.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.052+0000 7fdaa4f5d700 1 -- 172.21.15.72:0/162848864 shutdown_connections 2024-03-26T02:02:01.053 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:01.052+0000 7fdaa4f5d700 1 -- 172.21.15.72:0/162848864 wait complete. 2024-03-26T02:02:01.312 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:00 smithi173 ceph-mon[33839]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T02:02:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:01 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:02:02.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:01 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:02:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:01 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:02:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:02 smithi072 ceph-mon[30177]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T02:02:03.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:02 smithi072 ceph-mon[41619]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T02:02:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:02 smithi173 ceph-mon[33839]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T02:02:03.949 INFO:teuthology.orchestra.run.smithi072.stdout:163208757272 2024-03-26T02:02:03.949 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-26T02:02:05.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:04 smithi072 ceph-mon[41619]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T02:02:05.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:05 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:02:05.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:04 smithi072 ceph-mon[30177]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T02:02:05.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:05 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:02:05.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:02:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:02:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:02:05.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:04 smithi173 ceph-mon[33839]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T02:02:05.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:04 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:02:05.651 INFO:teuthology.orchestra.run.smithi072.stdout:107374182436 2024-03-26T02:02:05.651 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-26T02:02:05.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.285+0000 7f82ffb0b700 1 -- 172.21.15.72:0/2547078448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82f8106260 msgr2=0x7f82f810de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:05.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.285+0000 7f82ffb0b700 1 --2- 172.21.15.72:0/2547078448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82f8106260 0x7f82f810de30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f82f000b0a0 tx=0x7f82f000b3b0 comp rx=0 tx=0).stop 2024-03-26T02:02:05.711 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.285+0000 7f82ffb0b700 1 -- 172.21.15.72:0/2547078448 shutdown_connections 2024-03-26T02:02:05.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.285+0000 7f82ffb0b700 1 --2- 172.21.15.72:0/2547078448 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82f8106260 0x7f82f810de30 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:05.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.285+0000 7f82ffb0b700 1 --2- 172.21.15.72:0/2547078448 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82f8101210 0x7f82f81015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:05.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.285+0000 7f82ffb0b700 1 --2- 172.21.15.72:0/2547078448 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82f8101bc0 0x7f82f8105b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:05.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.285+0000 7f82ffb0b700 1 -- 172.21.15.72:0/2547078448 >> 172.21.15.72:0/2547078448 conn(0x7f82f8076fd0 msgr2=0x7f82f80773e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:05.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.285+0000 7f82ffb0b700 1 -- 172.21.15.72:0/2547078448 shutdown_connections 2024-03-26T02:02:05.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.285+0000 7f82ffb0b700 1 -- 172.21.15.72:0/2547078448 wait complete. 2024-03-26T02:02:05.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.285+0000 7f82ffb0b700 1 Processor -- start 2024-03-26T02:02:05.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.285+0000 7f82ffb0b700 1 -- start start 2024-03-26T02:02:05.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.286+0000 7f82ffb0b700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82f8101210 0x7f82f8198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:05.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.286+0000 7f82ffb0b700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82f8101bc0 0x7f82f8199100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:05.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.286+0000 7f82ffb0b700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82f8106260 0x7f82f81a3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:05.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.286+0000 7f82ffb0b700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f81999a0 con 0x7f82f8101bc0 2024-03-26T02:02:05.712 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.286+0000 7f82ffb0b700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f81a4180 con 0x7f82f8106260 2024-03-26T02:02:05.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.286+0000 7f82ffb0b700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82f81a42c0 con 0x7f82f8101210 2024-03-26T02:02:05.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.286+0000 7f82fd8a7700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82f8101210 0x7f82f8198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:05.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.286+0000 7f82fd8a7700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82f8101210 0x7f82f8198bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:59940/0 (socket says 172.21.15.72:59940) 2024-03-26T02:02:05.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.286+0000 7f82fd8a7700 1 -- 172.21.15.72:0/3963942445 learned_addr learned my addr 172.21.15.72:0/3963942445 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:02:05.713 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.286+0000 7f82fd0a6700 1 --2- 172.21.15.72:0/3963942445 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82f8101bc0 0x7f82f8199100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:05.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.286+0000 7f82fe0a8700 1 --2- 172.21.15.72:0/3963942445 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82f8106260 0x7f82f81a3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:05.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.286+0000 7f82fd0a6700 1 -- 172.21.15.72:0/3963942445 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82f8101210 msgr2=0x7f82f8198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:05.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.286+0000 7f82fd0a6700 1 --2- 172.21.15.72:0/3963942445 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82f8101210 0x7f82f8198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:05.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.286+0000 7f82fd0a6700 1 -- 172.21.15.72:0/3963942445 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82f8106260 msgr2=0x7f82f81a3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:05.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.286+0000 7f82fd0a6700 1 --2- 172.21.15.72:0/3963942445 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82f8106260 0x7f82f81a3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:05.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.286+0000 7f82fd0a6700 1 -- 172.21.15.72:0/3963942445 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f0009d00 con 0x7f82f8101bc0 2024-03-26T02:02:05.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.286+0000 7f82fd8a7700 1 --2- 172.21.15.72:0/3963942445 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f82f8101210 0x7f82f8198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:02:05.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.286+0000 7f82fe0a8700 1 --2- 172.21.15.72:0/3963942445 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f82f8106260 0x7f82f81a3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T02:02:05.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.286+0000 7f82fd0a6700 1 --2- 172.21.15.72:0/3963942445 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82f8101bc0 0x7f82f8199100 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f82e8009d10 tx=0x7f82e800c5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:05.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.287+0000 7f82eeffd700 1 -- 172.21.15.72:0/3963942445 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e800cdd0 con 0x7f82f8101bc0 2024-03-26T02:02:05.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.287+0000 7f82ffb0b700 1 -- 172.21.15.72:0/3963942445 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f8196a10 con 0x7f82f8101bc0 2024-03-26T02:02:05.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.287+0000 7f82eeffd700 1 -- 172.21.15.72:0/3963942445 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f82e801bd00 con 0x7f82f8101bc0 2024-03-26T02:02:05.714 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.287+0000 7f82eeffd700 1 -- 172.21.15.72:0/3963942445 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e8004870 con 0x7f82f8101bc0 2024-03-26T02:02:05.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.287+0000 7f82ffb0b700 1 -- 172.21.15.72:0/3963942445 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f8196f10 con 0x7f82f8101bc0 2024-03-26T02:02:05.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.288+0000 7f82ffb0b700 1 -- 172.21.15.72:0/3963942445 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f82f81109f0 con 0x7f82f8101bc0 2024-03-26T02:02:05.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.289+0000 7f82eeffd700 1 -- 172.21.15.72:0/3963942445 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7f82e800f450 con 0x7f82f8101bc0 2024-03-26T02:02:05.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.289+0000 7f82eeffd700 1 --2- 172.21.15.72:0/3963942445 >> v2:172.21.15.72:6800/3246690770 conn(0x7f82e406da80 0x7f82e406ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:05.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.289+0000 7f82fd8a7700 1 --2- 172.21.15.72:0/3963942445 >> v2:172.21.15.72:6800/3246690770 conn(0x7f82e406da80 0x7f82e406ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:05.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.289+0000 7f82eeffd700 1 -- 172.21.15.72:0/3963942445 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6259+0+0 (secure 0 0 0) 0x7f82e8095910 con 0x7f82f8101bc0 2024-03-26T02:02:05.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.289+0000 7f82eeffd700 1 --2- 172.21.15.72:0/3963942445 >> v2:172.21.15.72:6801/1438405769 conn(0x7f82e4077320 0x7f82e40797c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:05.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.289+0000 7f82eeffd700 1 -- 172.21.15.72:0/3963942445 --> v2:172.21.15.72:6801/1438405769 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f82e4079e50 con 0x7f82e4077320 2024-03-26T02:02:05.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.289+0000 7f82eeffd700 1 -- 172.21.15.72:0/3963942445 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82e809a2c0 con 0x7f82f8101bc0 2024-03-26T02:02:05.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.289+0000 7f82fe0a8700 1 --2- 172.21.15.72:0/3963942445 >> v2:172.21.15.72:6801/1438405769 conn(0x7f82e4077320 0x7f82e40797c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:05.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.290+0000 7f82fd8a7700 1 --2- 172.21.15.72:0/3963942445 >> v2:172.21.15.72:6800/3246690770 conn(0x7f82e406da80 0x7f82e406ff40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f82f400f4d0 tx=0x7f82f4005f90 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:05.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.290+0000 7f82fe0a8700 1 --2- 172.21.15.72:0/3963942445 >> v2:172.21.15.72:6801/1438405769 conn(0x7f82e4077320 0x7f82e40797c0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:05.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.290+0000 7f82eeffd700 1 -- 172.21.15.72:0/3963942445 <== osd.0 v2:172.21.15.72:6801/1438405769 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f82e4079e50 con 0x7f82e4077320 2024-03-26T02:02:05.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.317+0000 7f82ffb0b700 1 -- 172.21.15.72:0/3963942445 --> v2:172.21.15.72:6801/1438405769 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f82f80ff880 con 0x7f82e4077320 2024-03-26T02:02:05.715 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.318+0000 7f82eeffd700 1 -- 172.21.15.72:0/3963942445 <== osd.0 v2:172.21.15.72:6801/1438405769 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f82f80ff880 con 0x7f82e4077320 2024-03-26T02:02:05.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.318+0000 7f82ffb0b700 1 -- 172.21.15.72:0/3963942445 >> v2:172.21.15.72:6801/1438405769 conn(0x7f82e4077320 msgr2=0x7f82e40797c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:05.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.318+0000 7f82ffb0b700 1 --2- 172.21.15.72:0/3963942445 >> v2:172.21.15.72:6801/1438405769 conn(0x7f82e4077320 0x7f82e40797c0 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:05.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.318+0000 7f82ffb0b700 1 -- 172.21.15.72:0/3963942445 >> v2:172.21.15.72:6800/3246690770 conn(0x7f82e406da80 msgr2=0x7f82e406ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:05.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.318+0000 7f82ffb0b700 1 --2- 172.21.15.72:0/3963942445 >> v2:172.21.15.72:6800/3246690770 conn(0x7f82e406da80 0x7f82e406ff40 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f82f400f4d0 tx=0x7f82f4005f90 comp rx=0 tx=0).stop 2024-03-26T02:02:05.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.318+0000 7f82ffb0b700 1 -- 172.21.15.72:0/3963942445 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82f8101bc0 msgr2=0x7f82f8199100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:05.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.318+0000 7f82ffb0b700 1 --2- 172.21.15.72:0/3963942445 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f82f8101bc0 0x7f82f8199100 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f82e8009d10 tx=0x7f82e800c5b0 comp rx=0 tx=0).stop 2024-03-26T02:02:05.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.318+0000 7f82fe0a8700 1 -- 172.21.15.72:0/3963942445 reap_dead start 2024-03-26T02:02:05.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.318+0000 7f82ffb0b700 1 -- 172.21.15.72:0/3963942445 shutdown_connections 2024-03-26T02:02:05.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.318+0000 7f82ffb0b700 1 -- 172.21.15.72:0/3963942445 >> 172.21.15.72:0/3963942445 conn(0x7f82f8076fd0 msgr2=0x7f82f80fef20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:05.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.318+0000 7f82ffb0b700 1 -- 172.21.15.72:0/3963942445 shutdown_connections 2024-03-26T02:02:05.716 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:00.318+0000 7f82ffb0b700 1 -- 172.21.15.72:0/3963942445 wait complete. 2024-03-26T02:02:06.294 INFO:teuthology.orchestra.run.smithi072.stdout:231928233997 2024-03-26T02:02:06.295 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-26T02:02:06.913 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:02:06.913 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:02:06.953 INFO:teuthology.orchestra.run.smithi072.stdout:34359738419 2024-03-26T02:02:06.953 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-26T02:02:07.200 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:06 smithi072 ceph-mon[41619]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T02:02:07.200 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:06 smithi072 ceph-mon[30177]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T02:02:07.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:06 smithi173 ceph-mon[33839]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T02:02:08.162 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:02:09.194 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:08 smithi072 ceph-mon[30177]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-26T02:02:09.194 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:08 smithi072 ceph-mon[41619]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-26T02:02:09.312 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:08 smithi173 ceph-mon[33839]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-26T02:02:10.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:02:10 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:02:10.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:10 smithi072 ceph-mon[41619]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:10.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:10 smithi072 ceph-mon[30177]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:11.085 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:10 smithi173 ceph-mon[33839]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:11.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.851+0000 7f980582f700 1 -- 172.21.15.72:0/2217671763 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9800101bc0 msgr2=0x7f9800105bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:11.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.851+0000 7f980582f700 1 --2- 172.21.15.72:0/2217671763 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9800101bc0 0x7f9800105bc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f97e8009990 tx=0x7f97e8009ca0 comp rx=0 tx=0).stop 2024-03-26T02:02:11.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.851+0000 7f980582f700 1 -- 172.21.15.72:0/2217671763 shutdown_connections 2024-03-26T02:02:11.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.851+0000 7f980582f700 1 --2- 172.21.15.72:0/2217671763 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f98001062a0 0x7f980010de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:11.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.851+0000 7f980582f700 1 --2- 172.21.15.72:0/2217671763 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9800101210 0x7f98001015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:11.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.851+0000 7f980582f700 1 --2- 172.21.15.72:0/2217671763 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9800101bc0 0x7f9800105bc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:11.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.851+0000 7f980582f700 1 -- 172.21.15.72:0/2217671763 >> 172.21.15.72:0/2217671763 conn(0x7f9800076fa0 msgr2=0x7f98000773b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:11.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.851+0000 7f980582f700 1 -- 172.21.15.72:0/2217671763 shutdown_connections 2024-03-26T02:02:11.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.851+0000 7f980582f700 1 -- 172.21.15.72:0/2217671763 wait complete. 2024-03-26T02:02:11.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.852+0000 7f980582f700 1 Processor -- start 2024-03-26T02:02:11.859 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.852+0000 7f980582f700 1 -- start start 2024-03-26T02:02:11.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.852+0000 7f980582f700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9800101210 0x7f9800198b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:11.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.852+0000 7f980582f700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9800101bc0 0x7f9800199070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:11.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.852+0000 7f980582f700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98001062a0 0x7f98001a3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:11.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.852+0000 7f980582f700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98001998e0 con 0x7f9800101210 2024-03-26T02:02:11.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.852+0000 7f980582f700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98001a40f0 con 0x7f98001062a0 2024-03-26T02:02:11.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.852+0000 7f980582f700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98001a4230 con 0x7f9800101bc0 2024-03-26T02:02:11.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.852+0000 7f97feffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9800101210 0x7f9800198b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:11.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.852+0000 7f97fe7fc700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9800101bc0 0x7f9800199070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:11.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.852+0000 7f97feffd700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9800101210 0x7f9800198b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33156/0 (socket says 172.21.15.72:33156) 2024-03-26T02:02:11.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.852+0000 7f97fe7fc700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9800101bc0 0x7f9800199070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:43540/0 (socket says 172.21.15.72:43540) 2024-03-26T02:02:11.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.852+0000 7f97feffd700 1 -- 172.21.15.72:0/433240979 learned_addr learned my addr 172.21.15.72:0/433240979 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:02:11.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.852+0000 7f97ff7fe700 1 --2- 172.21.15.72:0/433240979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98001062a0 0x7f98001a3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:11.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.853+0000 7f97fe7fc700 1 -- 172.21.15.72:0/433240979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98001062a0 msgr2=0x7f98001a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:11.860 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.853+0000 7f97fe7fc700 1 --2- 172.21.15.72:0/433240979 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f98001062a0 0x7f98001a3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:11.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.853+0000 7f97fe7fc700 1 -- 172.21.15.72:0/433240979 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9800101210 msgr2=0x7f9800198b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:11.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.853+0000 7f97fe7fc700 1 --2- 172.21.15.72:0/433240979 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9800101210 0x7f9800198b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:11.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.853+0000 7f97fe7fc700 1 -- 172.21.15.72:0/433240979 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f0009520 con 0x7f9800101bc0 2024-03-26T02:02:11.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.853+0000 7f97feffd700 1 --2- 172.21.15.72:0/433240979 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9800101210 0x7f9800198b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:02:11.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.853+0000 7f97fe7fc700 1 --2- 172.21.15.72:0/433240979 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9800101bc0 0x7f9800199070 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f97e80098e0 tx=0x7f97e8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:11.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.855+0000 7f980482d700 1 -- 172.21.15.72:0/433240979 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e801d070 con 0x7f9800101bc0 2024-03-26T02:02:11.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.855+0000 7f980482d700 1 -- 172.21.15.72:0/433240979 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f97e8022470 con 0x7f9800101bc0 2024-03-26T02:02:11.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.855+0000 7f980582f700 1 -- 172.21.15.72:0/433240979 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e80095f0 con 0x7f9800101bc0 2024-03-26T02:02:11.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.855+0000 7f980482d700 1 -- 172.21.15.72:0/433240979 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e8017650 con 0x7f9800101bc0 2024-03-26T02:02:11.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.855+0000 7f980582f700 1 -- 172.21.15.72:0/433240979 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98001a4740 con 0x7f9800101bc0 2024-03-26T02:02:11.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.856+0000 7f980582f700 1 -- 172.21.15.72:0/433240979 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f98001109c0 con 0x7f9800101bc0 2024-03-26T02:02:11.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.857+0000 7f980482d700 1 -- 172.21.15.72:0/433240979 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7f97e80177b0 con 0x7f9800101bc0 2024-03-26T02:02:11.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.857+0000 7f980482d700 1 --2- 172.21.15.72:0/433240979 >> v2:172.21.15.72:6800/3246690770 conn(0x7f97ec06da80 0x7f97ec06ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:11.861 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.857+0000 7f980482d700 1 -- 172.21.15.72:0/433240979 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6259+0+0 (secure 0 0 0) 0x7f97e808e690 con 0x7f9800101bc0 2024-03-26T02:02:11.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.857+0000 7f97feffd700 1 --2- 172.21.15.72:0/433240979 >> v2:172.21.15.72:6800/3246690770 conn(0x7f97ec06da80 0x7f97ec06ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:11.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.857+0000 7f980482d700 1 --2- 172.21.15.72:0/433240979 >> v2:172.21.15.72:6805/2073231730 conn(0x7f97ec0773a0 0x7f97ec079840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:11.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.857+0000 7f980482d700 1 -- 172.21.15.72:0/433240979 --> v2:172.21.15.72:6805/2073231730 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f97ec079ed0 con 0x7f97ec0773a0 2024-03-26T02:02:11.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.857+0000 7f980482d700 1 -- 172.21.15.72:0/433240979 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_get_version_reply(handle=1 version=71) v2 ==== 24+0+0 (secure 0 0 0) 0x7f97e8090c70 con 0x7f9800101bc0 2024-03-26T02:02:11.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.858+0000 7f97ff7fe700 1 --2- 172.21.15.72:0/433240979 >> v2:172.21.15.72:6805/2073231730 conn(0x7f97ec0773a0 0x7f97ec079840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:11.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.858+0000 7f97feffd700 1 --2- 172.21.15.72:0/433240979 >> v2:172.21.15.72:6800/3246690770 conn(0x7f97ec06da80 0x7f97ec06ff40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f97f8009fd0 tx=0x7f97f8009040 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:11.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.858+0000 7f97ff7fe700 1 --2- 172.21.15.72:0/433240979 >> v2:172.21.15.72:6805/2073231730 conn(0x7f97ec0773a0 0x7f97ec079840 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:11.862 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.859+0000 7f980482d700 1 -- 172.21.15.72:0/433240979 <== osd.1 v2:172.21.15.72:6805/2073231730 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f97ec079ed0 con 0x7f97ec0773a0 2024-03-26T02:02:11.875 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.875+0000 7f980582f700 1 -- 172.21.15.72:0/433240979 --> v2:172.21.15.72:6805/2073231730 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9800062710 con 0x7f97ec0773a0 2024-03-26T02:02:11.876 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.875+0000 7f980482d700 1 -- 172.21.15.72:0/433240979 <== osd.1 v2:172.21.15.72:6805/2073231730 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9800062710 con 0x7f97ec0773a0 2024-03-26T02:02:11.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.876+0000 7f980582f700 1 -- 172.21.15.72:0/433240979 >> v2:172.21.15.72:6805/2073231730 conn(0x7f97ec0773a0 msgr2=0x7f97ec079840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:11.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.876+0000 7f980582f700 1 --2- 172.21.15.72:0/433240979 >> v2:172.21.15.72:6805/2073231730 conn(0x7f97ec0773a0 0x7f97ec079840 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:11.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.876+0000 7f980582f700 1 -- 172.21.15.72:0/433240979 >> v2:172.21.15.72:6800/3246690770 conn(0x7f97ec06da80 msgr2=0x7f97ec06ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:11.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.876+0000 7f980582f700 1 --2- 172.21.15.72:0/433240979 >> v2:172.21.15.72:6800/3246690770 conn(0x7f97ec06da80 0x7f97ec06ff40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f97f8009fd0 tx=0x7f97f8009040 comp rx=0 tx=0).stop 2024-03-26T02:02:11.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.876+0000 7f980582f700 1 -- 172.21.15.72:0/433240979 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9800101bc0 msgr2=0x7f9800199070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:11.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.876+0000 7f980582f700 1 --2- 172.21.15.72:0/433240979 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9800101bc0 0x7f9800199070 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f97e80098e0 tx=0x7f97e8004730 comp rx=0 tx=0).stop 2024-03-26T02:02:11.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.876+0000 7f97ff7fe700 1 -- 172.21.15.72:0/433240979 reap_dead start 2024-03-26T02:02:11.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.876+0000 7f980582f700 1 -- 172.21.15.72:0/433240979 shutdown_connections 2024-03-26T02:02:11.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.876+0000 7f980582f700 1 -- 172.21.15.72:0/433240979 >> 172.21.15.72:0/433240979 conn(0x7f9800076fa0 msgr2=0x7f9800107420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:11.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.876+0000 7f980582f700 1 -- 172.21.15.72:0/433240979 shutdown_connections 2024-03-26T02:02:11.877 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:11.876+0000 7f980582f700 1 -- 172.21.15.72:0/433240979 wait complete. 2024-03-26T02:02:12.147 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:11 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:02:12.147 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:11 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:02:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:11 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:02:12.449 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:02:13.078 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:12 smithi072 ceph-mon[30177]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:13.078 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3459945211' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T02:02:13.078 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:12 smithi072 ceph-mon[41619]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:13.078 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3459945211' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T02:02:13.079 INFO:teuthology.orchestra.run.smithi072.stdout:55834574897 2024-03-26T02:02:13.080 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-26T02:02:13.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.061+0000 7f5e3c355700 1 -- 172.21.15.72:0/673145939 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5e34101030 msgr2=0x7f5e3410ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:13.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.061+0000 7f5e3c355700 1 --2- 172.21.15.72:0/673145939 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5e34101030 0x7f5e3410ef90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c00b0a0 tx=0x7f5e2c00b3b0 comp rx=0 tx=0).stop 2024-03-26T02:02:13.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.061+0000 7f5e3c355700 1 -- 172.21.15.72:0/673145939 shutdown_connections 2024-03-26T02:02:13.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.061+0000 7f5e3c355700 1 --2- 172.21.15.72:0/673145939 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5e34101030 0x7f5e3410ef90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:13.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.061+0000 7f5e3c355700 1 --2- 172.21.15.72:0/673145939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e341070f0 0x7f5e341074d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:13.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.061+0000 7f5e3c355700 1 --2- 172.21.15.72:0/673145939 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e34100670 0x7f5e34100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:13.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.061+0000 7f5e3c355700 1 -- 172.21.15.72:0/673145939 >> 172.21.15.72:0/673145939 conn(0x7f5e34077580 msgr2=0x7f5e340fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:13.216 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.062+0000 7f5e3c355700 1 -- 172.21.15.72:0/673145939 shutdown_connections 2024-03-26T02:02:13.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.062+0000 7f5e3c355700 1 -- 172.21.15.72:0/673145939 wait complete. 2024-03-26T02:02:13.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.062+0000 7f5e3c355700 1 Processor -- start 2024-03-26T02:02:13.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.062+0000 7f5e3c355700 1 -- start start 2024-03-26T02:02:13.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.062+0000 7f5e3c355700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e34100670 0x7f5e34197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:13.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.062+0000 7f5e3c355700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e34101030 0x7f5e341984e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:13.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.062+0000 7f5e3c355700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5e341070f0 0x7f5e3419c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:13.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.062+0000 7f5e3c355700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3419ce80 con 0x7f5e34100670 2024-03-26T02:02:13.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.062+0000 7f5e3c355700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3419cff0 con 0x7f5e34101030 2024-03-26T02:02:13.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.062+0000 7f5e3c355700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3419d1b0 con 0x7f5e341070f0 2024-03-26T02:02:13.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.062+0000 7f5e3a0f1700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e34100670 0x7f5e34197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:13.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.062+0000 7f5e3a8f2700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5e341070f0 0x7f5e3419c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:13.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.062+0000 7f5e3a8f2700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5e341070f0 0x7f5e3419c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:43554/0 (socket says 172.21.15.72:43554) 2024-03-26T02:02:13.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.062+0000 7f5e3a0f1700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e34100670 0x7f5e34197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33186/0 (socket says 172.21.15.72:33186) 2024-03-26T02:02:13.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.062+0000 7f5e3a8f2700 1 -- 172.21.15.72:0/3459945211 learned_addr learned my addr 172.21.15.72:0/3459945211 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:02:13.218 INFO:teuthology.orchestra.run.smithi072.stdout:197568495637 2024-03-26T02:02:13.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.063+0000 7f5e398f0700 1 --2- 172.21.15.72:0/3459945211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e34101030 0x7f5e341984e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:13.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.063+0000 7f5e3a8f2700 1 -- 172.21.15.72:0/3459945211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e34101030 msgr2=0x7f5e341984e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:13.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.063+0000 7f5e3a8f2700 1 --2- 172.21.15.72:0/3459945211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e34101030 0x7f5e341984e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:13.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.063+0000 7f5e3a8f2700 1 -- 172.21.15.72:0/3459945211 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e34100670 msgr2=0x7f5e34197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:13.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.063+0000 7f5e3a8f2700 1 --2- 172.21.15.72:0/3459945211 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e34100670 0x7f5e34197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:13.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.063+0000 7f5e3a8f2700 1 -- 172.21.15.72:0/3459945211 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e24009520 con 0x7f5e341070f0 2024-03-26T02:02:13.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.063+0000 7f5e3a0f1700 1 --2- 172.21.15.72:0/3459945211 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e34100670 0x7f5e34197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:02:13.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.063+0000 7f5e3a8f2700 1 --2- 172.21.15.72:0/3459945211 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5e341070f0 0x7f5e3419c870 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c00b070 tx=0x7f5e2c004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:13.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.063+0000 7f5e2b7fe700 1 -- 172.21.15.72:0/3459945211 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c00e040 con 0x7f5e341070f0 2024-03-26T02:02:13.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.063+0000 7f5e2b7fe700 1 -- 172.21.15.72:0/3459945211 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f5e2c009480 con 0x7f5e341070f0 2024-03-26T02:02:13.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.063+0000 7f5e3c355700 1 -- 172.21.15.72:0/3459945211 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e2c009d00 con 0x7f5e341070f0 2024-03-26T02:02:13.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.063+0000 7f5e2b7fe700 1 -- 172.21.15.72:0/3459945211 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c0052a0 con 0x7f5e341070f0 2024-03-26T02:02:13.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.063+0000 7f5e3c355700 1 -- 172.21.15.72:0/3459945211 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3419d6c0 con 0x7f5e341070f0 2024-03-26T02:02:13.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.064+0000 7f5e3c355700 1 -- 172.21.15.72:0/3459945211 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e34103990 con 0x7f5e341070f0 2024-03-26T02:02:13.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.064+0000 7f5e2b7fe700 1 -- 172.21.15.72:0/3459945211 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7f5e2c01d470 con 0x7f5e341070f0 2024-03-26T02:02:13.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.065+0000 7f5e2b7fe700 1 --2- 172.21.15.72:0/3459945211 >> v2:172.21.15.72:6800/3246690770 conn(0x7f5e2006da80 0x7f5e2006ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:13.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.065+0000 7f5e2b7fe700 1 -- 172.21.15.72:0/3459945211 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6259+0+0 (secure 0 0 0) 0x7f5e2c08e970 con 0x7f5e341070f0 2024-03-26T02:02:13.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.065+0000 7f5e3a0f1700 1 --2- 172.21.15.72:0/3459945211 >> v2:172.21.15.72:6800/3246690770 conn(0x7f5e2006da80 0x7f5e2006ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:13.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.065+0000 7f5e3a0f1700 1 --2- 172.21.15.72:0/3459945211 >> v2:172.21.15.72:6800/3246690770 conn(0x7f5e2006da80 0x7f5e2006ff40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5e30009de0 tx=0x7f5e30009190 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:13.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.070+0000 7f5e2b7fe700 1 -- 172.21.15.72:0/3459945211 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5e2c05bd90 con 0x7f5e341070f0 2024-03-26T02:02:13.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.272+0000 7f5e3c355700 1 -- 172.21.15.72:0/3459945211 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f5e34068d40 con 0x7f5e341070f0 2024-03-26T02:02:13.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.272+0000 7f5e2b7fe700 1 -- 172.21.15.72:0/3459945211 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f5e2c05b920 con 0x7f5e341070f0 2024-03-26T02:02:13.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.274+0000 7f5e3c355700 1 -- 172.21.15.72:0/3459945211 >> v2:172.21.15.72:6800/3246690770 conn(0x7f5e2006da80 msgr2=0x7f5e2006ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:13.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.274+0000 7f5e3c355700 1 --2- 172.21.15.72:0/3459945211 >> v2:172.21.15.72:6800/3246690770 conn(0x7f5e2006da80 0x7f5e2006ff40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5e30009de0 tx=0x7f5e30009190 comp rx=0 tx=0).stop 2024-03-26T02:02:13.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.274+0000 7f5e3c355700 1 -- 172.21.15.72:0/3459945211 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5e341070f0 msgr2=0x7f5e3419c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:13.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.274+0000 7f5e3c355700 1 --2- 172.21.15.72:0/3459945211 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5e341070f0 0x7f5e3419c870 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c00b070 tx=0x7f5e2c004d00 comp rx=0 tx=0).stop 2024-03-26T02:02:13.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.274+0000 7f5e3c355700 1 -- 172.21.15.72:0/3459945211 shutdown_connections 2024-03-26T02:02:13.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.274+0000 7f5e3c355700 1 --2- 172.21.15.72:0/3459945211 >> v2:172.21.15.72:6800/3246690770 conn(0x7f5e2006da80 0x7f5e2006ff40 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:13.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.274+0000 7f5e3c355700 1 --2- 172.21.15.72:0/3459945211 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5e341070f0 0x7f5e3419c870 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:13.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.274+0000 7f5e3c355700 1 --2- 172.21.15.72:0/3459945211 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5e34100670 0x7f5e34197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:13.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.274+0000 7f5e3c355700 1 --2- 172.21.15.72:0/3459945211 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5e34101030 0x7f5e341984e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:13.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.274+0000 7f5e3c355700 1 -- 172.21.15.72:0/3459945211 >> 172.21.15.72:0/3459945211 conn(0x7f5e34077580 msgr2=0x7f5e34104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:13.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.274+0000 7f5e3c355700 1 -- 172.21.15.72:0/3459945211 shutdown_connections 2024-03-26T02:02:13.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:12.274+0000 7f5e3c355700 1 -- 172.21.15.72:0/3459945211 wait complete. 2024-03-26T02:02:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:12 smithi173 ceph-mon[33839]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3459945211' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T02:02:13.794 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495633 got 197568495637 for osd.6 2024-03-26T02:02:13.794 DEBUG:teuthology.parallel:result is None 2024-03-26T02:02:14.471 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:02:15.105 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:14 smithi072 ceph-mon[30177]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:15.106 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:02:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:02:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:02:15.106 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:14 smithi072 ceph-mon[41619]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:15.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:14 smithi173 ceph-mon[33839]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:15.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.096+0000 7fbc7bf23700 1 -- 172.21.15.72:0/2496089715 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbc7406a7d0 msgr2=0x7fbc7406ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:15.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.096+0000 7fbc7bf23700 1 --2- 172.21.15.72:0/2496089715 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbc7406a7d0 0x7fbc7406ac50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fbc64009990 tx=0x7fbc64009ca0 comp rx=0 tx=0).stop 2024-03-26T02:02:15.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.096+0000 7fbc7bf23700 1 -- 172.21.15.72:0/2496089715 shutdown_connections 2024-03-26T02:02:15.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.096+0000 7fbc7bf23700 1 --2- 172.21.15.72:0/2496089715 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc7406b190 0x7fbc7410eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:15.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.096+0000 7fbc7bf23700 1 --2- 172.21.15.72:0/2496089715 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc741080b0 0x7fbc74108490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:15.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.096+0000 7fbc7bf23700 1 --2- 172.21.15.72:0/2496089715 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbc7406a7d0 0x7fbc7406ac50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:15.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.096+0000 7fbc7bf23700 1 -- 172.21.15.72:0/2496089715 >> 172.21.15.72:0/2496089715 conn(0x7fbc74077f90 msgr2=0x7fbc740783a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:15.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.096+0000 7fbc7bf23700 1 -- 172.21.15.72:0/2496089715 shutdown_connections 2024-03-26T02:02:15.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.096+0000 7fbc7bf23700 1 -- 172.21.15.72:0/2496089715 wait complete. 2024-03-26T02:02:15.625 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.097+0000 7fbc7bf23700 1 Processor -- start 2024-03-26T02:02:15.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.097+0000 7fbc7bf23700 1 -- start start 2024-03-26T02:02:15.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.097+0000 7fbc7bf23700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbc7406a7d0 0x7fbc74197d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:15.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.097+0000 7fbc7bf23700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc7406b190 0x7fbc74198260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:15.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.097+0000 7fbc7bf23700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc741080b0 0x7fbc7419c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:15.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.097+0000 7fbc7bf23700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7419cc00 con 0x7fbc7406b190 2024-03-26T02:02:15.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.097+0000 7fbc7bf23700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7419cd70 con 0x7fbc741080b0 2024-03-26T02:02:15.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.097+0000 7fbc7bf23700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc7419cf30 con 0x7fbc7406a7d0 2024-03-26T02:02:15.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.097+0000 7fbc79cbf700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbc7406a7d0 0x7fbc74197d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:15.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.097+0000 7fbc794be700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc7406b190 0x7fbc74198260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:15.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.097+0000 7fbc79cbf700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbc7406a7d0 0x7fbc74197d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:43574/0 (socket says 172.21.15.72:43574) 2024-03-26T02:02:15.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.097+0000 7fbc79cbf700 1 -- 172.21.15.72:0/4286815095 learned_addr learned my addr 172.21.15.72:0/4286815095 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:02:15.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.097+0000 7fbc7a4c0700 1 --2- 172.21.15.72:0/4286815095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc741080b0 0x7fbc7419c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:15.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.097+0000 7fbc794be700 1 -- 172.21.15.72:0/4286815095 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbc7406a7d0 msgr2=0x7fbc74197d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:15.626 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.097+0000 7fbc794be700 1 --2- 172.21.15.72:0/4286815095 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbc7406a7d0 0x7fbc74197d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:15.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.097+0000 7fbc794be700 1 -- 172.21.15.72:0/4286815095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc741080b0 msgr2=0x7fbc7419c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:15.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.097+0000 7fbc794be700 1 --2- 172.21.15.72:0/4286815095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc741080b0 0x7fbc7419c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:15.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.097+0000 7fbc794be700 1 -- 172.21.15.72:0/4286815095 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc6c009c40 con 0x7fbc7406b190 2024-03-26T02:02:15.627 INFO:teuthology.orchestra.run.smithi072.stdout:77309411371 2024-03-26T02:02:15.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.097+0000 7fbc79cbf700 1 --2- 172.21.15.72:0/4286815095 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbc7406a7d0 0x7fbc74197d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:02:15.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.097+0000 7fbc794be700 1 --2- 172.21.15.72:0/4286815095 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc7406b190 0x7fbc74198260 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbc640059d0 tx=0x7fbc64004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:15.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.097+0000 7fbc6affd700 1 -- 172.21.15.72:0/4286815095 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc6401d070 con 0x7fbc7406b190 2024-03-26T02:02:15.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.097+0000 7fbc6affd700 1 -- 172.21.15.72:0/4286815095 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fbc64022470 con 0x7fbc7406b190 2024-03-26T02:02:15.627 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.098+0000 7fbc7bf23700 1 -- 172.21.15.72:0/4286815095 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc640095f0 con 0x7fbc7406b190 2024-03-26T02:02:15.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.098+0000 7fbc6affd700 1 -- 172.21.15.72:0/4286815095 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc64017650 con 0x7fbc7406b190 2024-03-26T02:02:15.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.098+0000 7fbc7bf23700 1 -- 172.21.15.72:0/4286815095 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc7419d430 con 0x7fbc7406b190 2024-03-26T02:02:15.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.099+0000 7fbc7bf23700 1 -- 172.21.15.72:0/4286815095 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc74068d40 con 0x7fbc7406b190 2024-03-26T02:02:15.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.099+0000 7fbc6affd700 1 -- 172.21.15.72:0/4286815095 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7fbc64022900 con 0x7fbc7406b190 2024-03-26T02:02:15.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.099+0000 7fbc6affd700 1 --2- 172.21.15.72:0/4286815095 >> v2:172.21.15.72:6800/3246690770 conn(0x7fbc6006da80 0x7fbc6006ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:15.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.100+0000 7fbc6affd700 1 -- 172.21.15.72:0/4286815095 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6259+0+0 (secure 0 0 0) 0x7fbc6408e8e0 con 0x7fbc7406b190 2024-03-26T02:02:15.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.100+0000 7fbc79cbf700 1 --2- 172.21.15.72:0/4286815095 >> v2:172.21.15.72:6800/3246690770 conn(0x7fbc6006da80 0x7fbc6006ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:15.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.100+0000 7fbc79cbf700 1 --2- 172.21.15.72:0/4286815095 >> v2:172.21.15.72:6800/3246690770 conn(0x7fbc6006da80 0x7fbc6006ff40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbc70009ea0 tx=0x7fbc70009290 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:15.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.104+0000 7fbc6affd700 1 -- 172.21.15.72:0/4286815095 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fbc6405bd00 con 0x7fbc7406b190 2024-03-26T02:02:15.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.305+0000 7fbc7bf23700 1 -- 172.21.15.72:0/4286815095 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fbc741117e0 con 0x7fbc7406b190 2024-03-26T02:02:15.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.305+0000 7fbc6affd700 1 -- 172.21.15.72:0/4286815095 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbc6405b890 con 0x7fbc7406b190 2024-03-26T02:02:15.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.307+0000 7fbc7bf23700 1 -- 172.21.15.72:0/4286815095 >> v2:172.21.15.72:6800/3246690770 conn(0x7fbc6006da80 msgr2=0x7fbc6006ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:15.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.307+0000 7fbc7bf23700 1 --2- 172.21.15.72:0/4286815095 >> v2:172.21.15.72:6800/3246690770 conn(0x7fbc6006da80 0x7fbc6006ff40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fbc70009ea0 tx=0x7fbc70009290 comp rx=0 tx=0).stop 2024-03-26T02:02:15.628 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.307+0000 7fbc7bf23700 1 -- 172.21.15.72:0/4286815095 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc7406b190 msgr2=0x7fbc74198260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:15.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.307+0000 7fbc7bf23700 1 --2- 172.21.15.72:0/4286815095 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc7406b190 0x7fbc74198260 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fbc640059d0 tx=0x7fbc64004730 comp rx=0 tx=0).stop 2024-03-26T02:02:15.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.307+0000 7fbc7bf23700 1 -- 172.21.15.72:0/4286815095 shutdown_connections 2024-03-26T02:02:15.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.307+0000 7fbc7bf23700 1 --2- 172.21.15.72:0/4286815095 >> v2:172.21.15.72:6800/3246690770 conn(0x7fbc6006da80 0x7fbc6006ff40 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:15.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.307+0000 7fbc7bf23700 1 --2- 172.21.15.72:0/4286815095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbc741080b0 0x7fbc7419c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:15.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.307+0000 7fbc7bf23700 1 --2- 172.21.15.72:0/4286815095 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbc7406a7d0 0x7fbc74197d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:15.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.307+0000 7fbc7bf23700 1 --2- 172.21.15.72:0/4286815095 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbc7406b190 0x7fbc74198260 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:15.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.307+0000 7fbc7bf23700 1 -- 172.21.15.72:0/4286815095 >> 172.21.15.72:0/4286815095 conn(0x7fbc74077f90 msgr2=0x7fbc74105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:15.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.307+0000 7fbc7bf23700 1 -- 172.21.15.72:0/4286815095 shutdown_connections 2024-03-26T02:02:15.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:15.307+0000 7fbc7bf23700 1 -- 172.21.15.72:0/4286815095 wait complete. 2024-03-26T02:02:15.800 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:02:15.801 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:02:15.801 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:02:15.906 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4286815095' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T02:02:15.907 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4286815095' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T02:02:16.211 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411368 got 77309411371 for osd.2 2024-03-26T02:02:16.211 DEBUG:teuthology.parallel:result is None 2024-03-26T02:02:16.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4286815095' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T02:02:16.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:16 smithi072 ceph-mon[30177]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:16.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:16 smithi072 ceph-mon[41619]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:17.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:16 smithi173 ceph-mon[33839]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:18.489 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.374+0000 7fb309e84700 1 -- 172.21.15.72:0/3286098269 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb304106790 msgr2=0x7fb304106b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:18.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.374+0000 7fb309e84700 1 --2- 172.21.15.72:0/3286098269 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb304106790 0x7fb304106b70 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc00b0d0 tx=0x7fb2fc00b3e0 comp rx=0 tx=0).stop 2024-03-26T02:02:18.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.374+0000 7fb309e84700 1 -- 172.21.15.72:0/3286098269 shutdown_connections 2024-03-26T02:02:18.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.374+0000 7fb309e84700 1 --2- 172.21.15.72:0/3286098269 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb304101370 0x7fb30410d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:18.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.374+0000 7fb309e84700 1 --2- 172.21.15.72:0/3286098269 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb304106790 0x7fb304106b70 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:18.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.374+0000 7fb309e84700 1 --2- 172.21.15.72:0/3286098269 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb3041009b0 0x7fb304100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:18.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.374+0000 7fb309e84700 1 -- 172.21.15.72:0/3286098269 >> 172.21.15.72:0/3286098269 conn(0x7fb3040762b0 msgr2=0x7fb3040766c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:18.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.374+0000 7fb309e84700 1 -- 172.21.15.72:0/3286098269 shutdown_connections 2024-03-26T02:02:18.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.374+0000 7fb309e84700 1 -- 172.21.15.72:0/3286098269 wait complete. 2024-03-26T02:02:18.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.375+0000 7fb309e84700 1 Processor -- start 2024-03-26T02:02:18.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.375+0000 7fb309e84700 1 -- start start 2024-03-26T02:02:18.490 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.375+0000 7fb309e84700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3041009b0 0x7fb30419c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:18.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.375+0000 7fb309e84700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb304101370 0x7fb30419ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:18.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.375+0000 7fb309e84700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb304106790 0x7fb304196b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:18.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.375+0000 7fb309e84700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb304197190 con 0x7fb304106790 2024-03-26T02:02:18.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.375+0000 7fb309e84700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3041972d0 con 0x7fb3041009b0 2024-03-26T02:02:18.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.375+0000 7fb309e84700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb304197490 con 0x7fb304101370 2024-03-26T02:02:18.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.375+0000 7fb302ffd700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb304101370 0x7fb30419ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:18.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.375+0000 7fb302ffd700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb304101370 0x7fb30419ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:43588/0 (socket says 172.21.15.72:43588) 2024-03-26T02:02:18.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.375+0000 7fb302ffd700 1 -- 172.21.15.72:0/2721813957 learned_addr learned my addr 172.21.15.72:0/2721813957 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:02:18.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.375+0000 7fb303fff700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb304106790 0x7fb304196b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:18.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.375+0000 7fb3037fe700 1 --2- 172.21.15.72:0/2721813957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3041009b0 0x7fb30419c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:18.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.375+0000 7fb303fff700 1 -- 172.21.15.72:0/2721813957 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb304101370 msgr2=0x7fb30419ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:18.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.375+0000 7fb303fff700 1 --2- 172.21.15.72:0/2721813957 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb304101370 0x7fb30419ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:18.491 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.375+0000 7fb303fff700 1 -- 172.21.15.72:0/2721813957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3041009b0 msgr2=0x7fb30419c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:18.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.375+0000 7fb303fff700 1 --2- 172.21.15.72:0/2721813957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3041009b0 0x7fb30419c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:18.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.376+0000 7fb303fff700 1 -- 172.21.15.72:0/2721813957 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2ec009520 con 0x7fb304106790 2024-03-26T02:02:18.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.376+0000 7fb302ffd700 1 --2- 172.21.15.72:0/2721813957 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb304101370 0x7fb30419ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:02:18.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.376+0000 7fb303fff700 1 --2- 172.21.15.72:0/2721813957 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb304106790 0x7fb304196b30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb2f400b3f0 tx=0x7fb2f400b700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:18.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.376+0000 7fb300ff9700 1 -- 172.21.15.72:0/2721813957 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f4011830 con 0x7fb304106790 2024-03-26T02:02:18.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.376+0000 7fb309e84700 1 -- 172.21.15.72:0/2721813957 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2fc009d00 con 0x7fb304106790 2024-03-26T02:02:18.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.376+0000 7fb300ff9700 1 -- 172.21.15.72:0/2721813957 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fb2f4011e40 con 0x7fb304106790 2024-03-26T02:02:18.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.376+0000 7fb300ff9700 1 -- 172.21.15.72:0/2721813957 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2f4010d30 con 0x7fb304106790 2024-03-26T02:02:18.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.376+0000 7fb309e84700 1 -- 172.21.15.72:0/2721813957 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3040fe8a0 con 0x7fb304106790 2024-03-26T02:02:18.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.377+0000 7fb309e84700 1 -- 172.21.15.72:0/2721813957 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb304068d40 con 0x7fb304106790 2024-03-26T02:02:18.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.377+0000 7fb300ff9700 1 -- 172.21.15.72:0/2721813957 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7fb2f4011990 con 0x7fb304106790 2024-03-26T02:02:18.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.378+0000 7fb300ff9700 1 --2- 172.21.15.72:0/2721813957 >> v2:172.21.15.72:6800/3246690770 conn(0x7fb2f006da80 0x7fb2f006ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:18.492 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.378+0000 7fb300ff9700 1 -- 172.21.15.72:0/2721813957 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6259+0+0 (secure 0 0 0) 0x7fb2f408cda0 con 0x7fb304106790 2024-03-26T02:02:18.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.378+0000 7fb3037fe700 1 --2- 172.21.15.72:0/2721813957 >> v2:172.21.15.72:6800/3246690770 conn(0x7fb2f006da80 0x7fb2f006ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:18.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.378+0000 7fb3037fe700 1 --2- 172.21.15.72:0/2721813957 >> v2:172.21.15.72:6800/3246690770 conn(0x7fb2f006da80 0x7fb2f006ff40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc00b770 tx=0x7fb2fc00b040 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:18.493 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.383+0000 7fb300ff9700 1 -- 172.21.15.72:0/2721813957 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb2f405a1c0 con 0x7fb304106790 2024-03-26T02:02:18.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.589+0000 7fb309e84700 1 -- 172.21.15.72:0/2721813957 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fb30410fef0 con 0x7fb304106790 2024-03-26T02:02:18.590 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.589+0000 7fb300ff9700 1 -- 172.21.15.72:0/2721813957 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fb2f4010ed0 con 0x7fb304106790 2024-03-26T02:02:18.590 INFO:teuthology.orchestra.run.smithi072.stdout:107374182439 2024-03-26T02:02:18.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.591+0000 7fb309e84700 1 -- 172.21.15.72:0/2721813957 >> v2:172.21.15.72:6800/3246690770 conn(0x7fb2f006da80 msgr2=0x7fb2f006ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:18.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.591+0000 7fb309e84700 1 --2- 172.21.15.72:0/2721813957 >> v2:172.21.15.72:6800/3246690770 conn(0x7fb2f006da80 0x7fb2f006ff40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc00b770 tx=0x7fb2fc00b040 comp rx=0 tx=0).stop 2024-03-26T02:02:18.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.591+0000 7fb309e84700 1 -- 172.21.15.72:0/2721813957 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb304106790 msgr2=0x7fb304196b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:18.592 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.591+0000 7fb309e84700 1 --2- 172.21.15.72:0/2721813957 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb304106790 0x7fb304196b30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb2f400b3f0 tx=0x7fb2f400b700 comp rx=0 tx=0).stop 2024-03-26T02:02:18.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.592+0000 7fb309e84700 1 -- 172.21.15.72:0/2721813957 shutdown_connections 2024-03-26T02:02:18.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.592+0000 7fb309e84700 1 --2- 172.21.15.72:0/2721813957 >> v2:172.21.15.72:6800/3246690770 conn(0x7fb2f006da80 0x7fb2f006ff40 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:18.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.592+0000 7fb309e84700 1 --2- 172.21.15.72:0/2721813957 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb304106790 0x7fb304196b30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:18.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.592+0000 7fb309e84700 1 --2- 172.21.15.72:0/2721813957 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb3041009b0 0x7fb30419c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:18.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.592+0000 7fb309e84700 1 --2- 172.21.15.72:0/2721813957 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb304101370 0x7fb30419ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:18.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.592+0000 7fb309e84700 1 -- 172.21.15.72:0/2721813957 >> 172.21.15.72:0/2721813957 conn(0x7fb3040762b0 msgr2=0x7fb3040ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:18.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.592+0000 7fb309e84700 1 -- 172.21.15.72:0/2721813957 shutdown_connections 2024-03-26T02:02:18.593 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:18.592+0000 7fb309e84700 1 -- 172.21.15.72:0/2721813957 wait complete. 2024-03-26T02:02:19.165 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:18 smithi072 ceph-mon[30177]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:19.166 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2721813957' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T02:02:19.166 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:18 smithi072 ceph-mon[41619]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:19.166 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2721813957' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T02:02:19.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:18 smithi173 ceph-mon[33839]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:19.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2721813957' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T02:02:20.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:19 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:02:20.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2641298570' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T02:02:20.124 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:19 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:02:20.124 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2641298570' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T02:02:20.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.144+0000 7fd108efb700 1 -- 172.21.15.72:0/1694945462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd104101440 msgr2=0x7fd10410efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:20.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.144+0000 7fd108efb700 1 --2- 172.21.15.72:0/1694945462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd104101440 0x7fd10410efd0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc009fc0 tx=0x7fd0fc00b330 comp rx=0 tx=0).stop 2024-03-26T02:02:20.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.144+0000 7fd108efb700 1 -- 172.21.15.72:0/1694945462 shutdown_connections 2024-03-26T02:02:20.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.144+0000 7fd108efb700 1 --2- 172.21.15.72:0/1694945462 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd104101440 0x7fd10410efd0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:20.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.144+0000 7fd108efb700 1 --2- 172.21.15.72:0/1694945462 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd104107150 0x7fd104107530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:20.145 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.144+0000 7fd108efb700 1 --2- 172.21.15.72:0/1694945462 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd104100a80 0x7fd104100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:20.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.144+0000 7fd108efb700 1 -- 172.21.15.72:0/1694945462 >> 172.21.15.72:0/1694945462 conn(0x7fd1040762d0 msgr2=0x7fd1040766e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:20.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.145+0000 7fd108efb700 1 -- 172.21.15.72:0/1694945462 shutdown_connections 2024-03-26T02:02:20.146 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.145+0000 7fd108efb700 1 -- 172.21.15.72:0/1694945462 wait complete. 2024-03-26T02:02:20.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.145+0000 7fd108efb700 1 Processor -- start 2024-03-26T02:02:20.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.145+0000 7fd108efb700 1 -- start start 2024-03-26T02:02:20.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.145+0000 7fd108efb700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd104100a80 0x7fd104197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:20.147 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.145+0000 7fd108efb700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd104101440 0x7fd104198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:20.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.145+0000 7fd108efb700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd104107150 0x7fd10419c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:20.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.145+0000 7fd108efb700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10419cce0 con 0x7fd104100a80 2024-03-26T02:02:20.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.145+0000 7fd108efb700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10419ce50 con 0x7fd104107150 2024-03-26T02:02:20.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.145+0000 7fd108efb700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd10419d010 con 0x7fd104101440 2024-03-26T02:02:20.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.146+0000 7fd10259c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd104100a80 0x7fd104197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:20.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.146+0000 7fd101d9b700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd104101440 0x7fd104198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:20.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.146+0000 7fd10259c700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd104100a80 0x7fd104197e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33250/0 (socket says 172.21.15.72:33250) 2024-03-26T02:02:20.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.146+0000 7fd10259c700 1 -- 172.21.15.72:0/4070107562 learned_addr learned my addr 172.21.15.72:0/4070107562 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:02:20.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.146+0000 7fd102d9d700 1 --2- 172.21.15.72:0/4070107562 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd104107150 0x7fd10419c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:20.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.146+0000 7fd10259c700 1 -- 172.21.15.72:0/4070107562 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd104101440 msgr2=0x7fd104198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:20.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.146+0000 7fd10259c700 1 --2- 172.21.15.72:0/4070107562 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd104101440 0x7fd104198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:20.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.146+0000 7fd10259c700 1 -- 172.21.15.72:0/4070107562 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd104107150 msgr2=0x7fd10419c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:20.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.146+0000 7fd10259c700 1 --2- 172.21.15.72:0/4070107562 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd104107150 0x7fd10419c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:20.148 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.146+0000 7fd10259c700 1 -- 172.21.15.72:0/4070107562 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f80095f0 con 0x7fd104100a80 2024-03-26T02:02:20.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.146+0000 7fd102d9d700 1 --2- 172.21.15.72:0/4070107562 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd104107150 0x7fd10419c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T02:02:20.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.146+0000 7fd10259c700 1 --2- 172.21.15.72:0/4070107562 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd104100a80 0x7fd104197e50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd0f400b530 tx=0x7fd0f400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:20.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.146+0000 7fd0f37fe700 1 -- 172.21.15.72:0/4070107562 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f4011830 con 0x7fd104100a80 2024-03-26T02:02:20.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.146+0000 7fd108efb700 1 -- 172.21.15.72:0/4070107562 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0fc009c40 con 0x7fd104100a80 2024-03-26T02:02:20.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.146+0000 7fd0f37fe700 1 -- 172.21.15.72:0/4070107562 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fd0f4011e40 con 0x7fd104100a80 2024-03-26T02:02:20.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.146+0000 7fd0f37fe700 1 -- 172.21.15.72:0/4070107562 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f400f2e0 con 0x7fd104100a80 2024-03-26T02:02:20.149 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.146+0000 7fd108efb700 1 -- 172.21.15.72:0/4070107562 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1041a4980 con 0x7fd104100a80 2024-03-26T02:02:20.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.147+0000 7fd108efb700 1 -- 172.21.15.72:0/4070107562 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd104068d40 con 0x7fd104100a80 2024-03-26T02:02:20.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.148+0000 7fd0f37fe700 1 -- 172.21.15.72:0/4070107562 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7fd0f4011990 con 0x7fd104100a80 2024-03-26T02:02:20.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.148+0000 7fd0f37fe700 1 --2- 172.21.15.72:0/4070107562 >> v2:172.21.15.72:6800/3246690770 conn(0x7fd0ec06da80 0x7fd0ec06ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:20.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.148+0000 7fd0f37fe700 1 -- 172.21.15.72:0/4070107562 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6259+0+0 (secure 0 0 0) 0x7fd0f408cf80 con 0x7fd104100a80 2024-03-26T02:02:20.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.148+0000 7fd101d9b700 1 --2- 172.21.15.72:0/4070107562 >> v2:172.21.15.72:6800/3246690770 conn(0x7fd0ec06da80 0x7fd0ec06ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:20.150 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.149+0000 7fd101d9b700 1 --2- 172.21.15.72:0/4070107562 >> v2:172.21.15.72:6800/3246690770 conn(0x7fd0ec06da80 0x7fd0ec06ff40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd104112360 tx=0x7fd0f8009310 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:20.153 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.153+0000 7fd0f37fe700 1 -- 172.21.15.72:0/4070107562 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fd0f40c1280 con 0x7fd104100a80 2024-03-26T02:02:20.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.595+0000 7f4cf780a700 1 -- 172.21.15.72:0/3449921677 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4cf01019e0 msgr2=0x7f4cf0101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:20.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.595+0000 7f4cf780a700 1 --2- 172.21.15.72:0/3449921677 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4cf01019e0 0x7f4cf0101dc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4cec009990 tx=0x7f4cec009ca0 comp rx=0 tx=0).stop 2024-03-26T02:02:20.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.595+0000 7f4cf780a700 1 -- 172.21.15.72:0/3449921677 shutdown_connections 2024-03-26T02:02:20.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.595+0000 7f4cf780a700 1 --2- 172.21.15.72:0/3449921677 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cf0106300 0x7f4cf010ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:20.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.595+0000 7f4cf780a700 1 --2- 172.21.15.72:0/3449921677 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4cf01019e0 0x7f4cf0101dc0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:20.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.595+0000 7f4cf780a700 1 --2- 172.21.15.72:0/3449921677 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4cf0102390 0x7f4cf0105c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:20.217 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.595+0000 7f4cf780a700 1 -- 172.21.15.72:0/3449921677 >> 172.21.15.72:0/3449921677 conn(0x7f4cf0077580 msgr2=0x7f4cf00fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:20.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.595+0000 7f4cf780a700 1 -- 172.21.15.72:0/3449921677 shutdown_connections 2024-03-26T02:02:20.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.595+0000 7f4cf780a700 1 -- 172.21.15.72:0/3449921677 wait complete. 2024-03-26T02:02:20.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.595+0000 7f4cf780a700 1 Processor -- start 2024-03-26T02:02:20.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.595+0000 7f4cf780a700 1 -- start start 2024-03-26T02:02:20.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.596+0000 7f4cf780a700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4cf01019e0 0x7f4cf0199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:20.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.596+0000 7f4cf780a700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cf0102390 0x7f4cf019a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:20.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.596+0000 7f4cf780a700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4cf0106300 0x7f4cf019e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:20.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.596+0000 7f4cf780a700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf019ee30 con 0x7f4cf01019e0 2024-03-26T02:02:20.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.596+0000 7f4cf780a700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf019efa0 con 0x7f4cf0102390 2024-03-26T02:02:20.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.596+0000 7f4cf780a700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf019f160 con 0x7f4cf0106300 2024-03-26T02:02:20.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.596+0000 7f4cf55a6700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4cf01019e0 0x7f4cf0199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:20.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.596+0000 7f4cf55a6700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4cf01019e0 0x7f4cf0199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33232/0 (socket says 172.21.15.72:33232) 2024-03-26T02:02:20.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.596+0000 7f4cf55a6700 1 -- 172.21.15.72:0/2641298570 learned_addr learned my addr 172.21.15.72:0/2641298570 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:02:20.218 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.596+0000 7f4cf5da7700 1 --2- 172.21.15.72:0/2641298570 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4cf0106300 0x7f4cf019e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:20.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.596+0000 7f4cf4da5700 1 --2- 172.21.15.72:0/2641298570 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cf0102390 0x7f4cf019a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:20.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.596+0000 7f4cf55a6700 1 -- 172.21.15.72:0/2641298570 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4cf0106300 msgr2=0x7f4cf019e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:20.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.596+0000 7f4cf55a6700 1 --2- 172.21.15.72:0/2641298570 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4cf0106300 0x7f4cf019e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:20.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.596+0000 7f4cf55a6700 1 -- 172.21.15.72:0/2641298570 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cf0102390 msgr2=0x7f4cf019a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:20.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.596+0000 7f4cf55a6700 1 --2- 172.21.15.72:0/2641298570 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cf0102390 0x7f4cf019a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:20.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.596+0000 7f4cf55a6700 1 -- 172.21.15.72:0/2641298570 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce0009520 con 0x7f4cf01019e0 2024-03-26T02:02:20.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.596+0000 7f4cf5da7700 1 --2- 172.21.15.72:0/2641298570 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4cf0106300 0x7f4cf019e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:02:20.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.596+0000 7f4cf55a6700 1 --2- 172.21.15.72:0/2641298570 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4cf01019e0 0x7f4cf0199f50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4cec000c00 tx=0x7f4cec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:20.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.596+0000 7f4ce67fc700 1 -- 172.21.15.72:0/2641298570 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec01d070 con 0x7f4cf01019e0 2024-03-26T02:02:20.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.597+0000 7f4ce67fc700 1 -- 172.21.15.72:0/2641298570 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f4cec022470 con 0x7f4cf01019e0 2024-03-26T02:02:20.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.597+0000 7f4cf780a700 1 -- 172.21.15.72:0/2641298570 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cec0095f0 con 0x7f4cf01019e0 2024-03-26T02:02:20.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.597+0000 7f4ce67fc700 1 -- 172.21.15.72:0/2641298570 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cec017650 con 0x7f4cf01019e0 2024-03-26T02:02:20.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.597+0000 7f4cf780a700 1 -- 172.21.15.72:0/2641298570 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cf019f660 con 0x7f4cf01019e0 2024-03-26T02:02:20.219 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.597+0000 7f4cf780a700 1 -- 172.21.15.72:0/2641298570 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cf0102790 con 0x7f4cf01019e0 2024-03-26T02:02:20.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.598+0000 7f4ce67fc700 1 -- 172.21.15.72:0/2641298570 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7f4cec00fa50 con 0x7f4cf01019e0 2024-03-26T02:02:20.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.598+0000 7f4ce67fc700 1 --2- 172.21.15.72:0/2641298570 >> v2:172.21.15.72:6800/3246690770 conn(0x7f4cdc06da80 0x7f4cdc06ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:20.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.599+0000 7f4ce67fc700 1 -- 172.21.15.72:0/2641298570 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6259+0+0 (secure 0 0 0) 0x7f4cec08e9c0 con 0x7f4cf01019e0 2024-03-26T02:02:20.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.599+0000 7f4cf4da5700 1 --2- 172.21.15.72:0/2641298570 >> v2:172.21.15.72:6800/3246690770 conn(0x7f4cdc06da80 0x7f4cdc06ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:20.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.599+0000 7f4cf4da5700 1 --2- 172.21.15.72:0/2641298570 >> v2:172.21.15.72:6800/3246690770 conn(0x7f4cdc06da80 0x7f4cdc06ff40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0105c90 tx=0x7f4ce0009260 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:20.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.604+0000 7f4ce67fc700 1 -- 172.21.15.72:0/2641298570 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f4cec05bd40 con 0x7f4cf01019e0 2024-03-26T02:02:20.220 INFO:teuthology.orchestra.run.smithi072.stdout:128849018914 2024-03-26T02:02:20.220 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.805+0000 7f4cf780a700 1 -- 172.21.15.72:0/2641298570 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f4cf0068d40 con 0x7f4cf01019e0 2024-03-26T02:02:20.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.806+0000 7f4ce67fc700 1 -- 172.21.15.72:0/2641298570 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f4cec05b8d0 con 0x7f4cf01019e0 2024-03-26T02:02:20.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.807+0000 7f4cf780a700 1 -- 172.21.15.72:0/2641298570 >> v2:172.21.15.72:6800/3246690770 conn(0x7f4cdc06da80 msgr2=0x7f4cdc06ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:20.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.807+0000 7f4cf780a700 1 --2- 172.21.15.72:0/2641298570 >> v2:172.21.15.72:6800/3246690770 conn(0x7f4cdc06da80 0x7f4cdc06ff40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4cf0105c90 tx=0x7f4ce0009260 comp rx=0 tx=0).stop 2024-03-26T02:02:20.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.807+0000 7f4cf780a700 1 -- 172.21.15.72:0/2641298570 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4cf01019e0 msgr2=0x7f4cf0199f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:20.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.807+0000 7f4cf780a700 1 --2- 172.21.15.72:0/2641298570 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4cf01019e0 0x7f4cf0199f50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4cec000c00 tx=0x7f4cec004730 comp rx=0 tx=0).stop 2024-03-26T02:02:20.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.808+0000 7f4cf780a700 1 -- 172.21.15.72:0/2641298570 shutdown_connections 2024-03-26T02:02:20.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.808+0000 7f4cf780a700 1 --2- 172.21.15.72:0/2641298570 >> v2:172.21.15.72:6800/3246690770 conn(0x7f4cdc06da80 0x7f4cdc06ff40 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:20.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.808+0000 7f4cf780a700 1 --2- 172.21.15.72:0/2641298570 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f4cf0106300 0x7f4cf019e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:20.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.808+0000 7f4cf780a700 1 --2- 172.21.15.72:0/2641298570 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f4cf01019e0 0x7f4cf0199f50 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:20.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.808+0000 7f4cf780a700 1 --2- 172.21.15.72:0/2641298570 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4cf0102390 0x7f4cf019a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:20.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.808+0000 7f4cf780a700 1 -- 172.21.15.72:0/2641298570 >> 172.21.15.72:0/2641298570 conn(0x7f4cf0077580 msgr2=0x7f4cf0104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:20.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.808+0000 7f4cf780a700 1 -- 172.21.15.72:0/2641298570 shutdown_connections 2024-03-26T02:02:20.221 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:19.808+0000 7f4cf780a700 1 -- 172.21.15.72:0/2641298570 wait complete. 2024-03-26T02:02:20.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:19 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:02:20.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2641298570' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T02:02:20.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.373+0000 7fd108efb700 1 -- 172.21.15.72:0/4070107562 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fd104111890 con 0x7fd104100a80 2024-03-26T02:02:20.374 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.374+0000 7fd0f37fe700 1 -- 172.21.15.72:0/4070107562 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fd0f405a3a0 con 0x7fd104100a80 2024-03-26T02:02:20.375 INFO:teuthology.orchestra.run.smithi072.stdout:231928234002 2024-03-26T02:02:20.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.377+0000 7fd108efb700 1 -- 172.21.15.72:0/4070107562 >> v2:172.21.15.72:6800/3246690770 conn(0x7fd0ec06da80 msgr2=0x7fd0ec06ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:20.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.377+0000 7fd108efb700 1 --2- 172.21.15.72:0/4070107562 >> v2:172.21.15.72:6800/3246690770 conn(0x7fd0ec06da80 0x7fd0ec06ff40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd104112360 tx=0x7fd0f8009310 comp rx=0 tx=0).stop 2024-03-26T02:02:20.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.377+0000 7fd108efb700 1 -- 172.21.15.72:0/4070107562 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd104100a80 msgr2=0x7fd104197e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:20.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.377+0000 7fd108efb700 1 --2- 172.21.15.72:0/4070107562 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd104100a80 0x7fd104197e50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd0f400b530 tx=0x7fd0f400b840 comp rx=0 tx=0).stop 2024-03-26T02:02:20.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.377+0000 7fd108efb700 1 -- 172.21.15.72:0/4070107562 shutdown_connections 2024-03-26T02:02:20.377 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.377+0000 7fd108efb700 1 --2- 172.21.15.72:0/4070107562 >> v2:172.21.15.72:6800/3246690770 conn(0x7fd0ec06da80 0x7fd0ec06ff40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:20.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.377+0000 7fd108efb700 1 --2- 172.21.15.72:0/4070107562 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd104107150 0x7fd10419c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:20.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.377+0000 7fd108efb700 1 --2- 172.21.15.72:0/4070107562 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fd104100a80 0x7fd104197e50 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:20.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.377+0000 7fd108efb700 1 --2- 172.21.15.72:0/4070107562 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fd104101440 0x7fd104198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:20.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.377+0000 7fd108efb700 1 -- 172.21.15.72:0/4070107562 >> 172.21.15.72:0/4070107562 conn(0x7fd1040762d0 msgr2=0x7fd104105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:20.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.377+0000 7fd108efb700 1 -- 172.21.15.72:0/4070107562 shutdown_connections 2024-03-26T02:02:20.378 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:20.377+0000 7fd108efb700 1 -- 172.21.15.72:0/4070107562 wait complete. 2024-03-26T02:02:20.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:02:20 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:02:20.930 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018910 got 128849018914 for osd.4 2024-03-26T02:02:20.930 DEBUG:teuthology.parallel:result is None 2024-03-26T02:02:21.134 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:20 smithi072 ceph-mon[30177]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:21.134 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4070107562' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T02:02:21.135 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:20 smithi072 ceph-mon[41619]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:21.135 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4070107562' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T02:02:21.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:20 smithi173 ceph-mon[33839]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:21.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4070107562' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T02:02:21.778 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182436 got 107374182439 for osd.3 2024-03-26T02:02:21.779 DEBUG:teuthology.parallel:result is None 2024-03-26T02:02:22.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:21 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:02:22.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:21 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:02:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:21 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:02:22.728 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233997 got 231928234002 for osd.7 2024-03-26T02:02:22.728 DEBUG:teuthology.parallel:result is None 2024-03-26T02:02:22.966 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:22 smithi072 ceph-mon[30177]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:22.966 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:22 smithi072 ceph-mon[41619]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:23.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.199+0000 7f5dd2457700 1 -- 172.21.15.72:0/3410267671 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dcc1022c0 msgr2=0x7f5dcc1026a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:23.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.199+0000 7f5dd2457700 1 --2- 172.21.15.72:0/3410267671 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dcc1022c0 0x7f5dcc1026a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0009990 tx=0x7f5dc0009ca0 comp rx=0 tx=0).stop 2024-03-26T02:02:23.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.199+0000 7f5dd2457700 1 -- 172.21.15.72:0/3410267671 shutdown_connections 2024-03-26T02:02:23.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.199+0000 7f5dd2457700 1 --2- 172.21.15.72:0/3410267671 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5dcc1073a0 0x7f5dcc10ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:23.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.199+0000 7f5dd2457700 1 --2- 172.21.15.72:0/3410267671 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dcc1022c0 0x7f5dcc1026a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:23.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.199+0000 7f5dd2457700 1 --2- 172.21.15.72:0/3410267671 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5dcc102c70 0x7f5dcc106cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:23.200 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.199+0000 7f5dd2457700 1 -- 172.21.15.72:0/3410267671 >> 172.21.15.72:0/3410267671 conn(0x7f5dcc0fdb30 msgr2=0x7f5dcc0fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:23.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.199+0000 7f5dd2457700 1 -- 172.21.15.72:0/3410267671 shutdown_connections 2024-03-26T02:02:23.201 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.199+0000 7f5dd2457700 1 -- 172.21.15.72:0/3410267671 wait complete. 2024-03-26T02:02:23.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.200+0000 7f5dd2457700 1 Processor -- start 2024-03-26T02:02:23.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.200+0000 7f5dd2457700 1 -- start start 2024-03-26T02:02:23.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.200+0000 7f5dd2457700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5dcc1022c0 0x7f5dcc19d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:23.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.200+0000 7f5dd2457700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5dcc102c70 0x7f5dcc19de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:23.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.200+0000 7f5dd2457700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dcc1073a0 0x7f5dcc197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:23.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.200+0000 7f5dd2457700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dcc198150 con 0x7f5dcc1073a0 2024-03-26T02:02:23.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.200+0000 7f5dd2457700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dcc198290 con 0x7f5dcc102c70 2024-03-26T02:02:23.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.200+0000 7f5dd2457700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dcc198450 con 0x7f5dcc1022c0 2024-03-26T02:02:23.203 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.201+0000 7f5dcbfff700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5dcc1022c0 0x7f5dcc19d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:23.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.201+0000 7f5dd09f4700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dcc1073a0 0x7f5dcc197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:23.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.201+0000 7f5dd09f4700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dcc1073a0 0x7f5dcc197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40064/0 (socket says 172.21.15.72:40064) 2024-03-26T02:02:23.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.201+0000 7f5dcbfff700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5dcc1022c0 0x7f5dcc19d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:35184/0 (socket says 172.21.15.72:35184) 2024-03-26T02:02:23.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.201+0000 7f5dd09f4700 1 -- 172.21.15.72:0/4258848216 learned_addr learned my addr 172.21.15.72:0/4258848216 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:02:23.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.201+0000 7f5dcb7fe700 1 --2- 172.21.15.72:0/4258848216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5dcc102c70 0x7f5dcc19de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:23.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.201+0000 7f5dd09f4700 1 -- 172.21.15.72:0/4258848216 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5dcc1022c0 msgr2=0x7f5dcc19d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:23.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.201+0000 7f5dd09f4700 1 --2- 172.21.15.72:0/4258848216 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5dcc1022c0 0x7f5dcc19d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:23.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.201+0000 7f5dd09f4700 1 -- 172.21.15.72:0/4258848216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5dcc102c70 msgr2=0x7f5dcc19de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:23.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.201+0000 7f5dd09f4700 1 --2- 172.21.15.72:0/4258848216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5dcc102c70 0x7f5dcc19de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:23.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.201+0000 7f5dd09f4700 1 -- 172.21.15.72:0/4258848216 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc4009c40 con 0x7f5dcc1073a0 2024-03-26T02:02:23.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.201+0000 7f5dcb7fe700 1 --2- 172.21.15.72:0/4258848216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5dcc102c70 0x7f5dcc19de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T02:02:23.204 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.201+0000 7f5dd09f4700 1 --2- 172.21.15.72:0/4258848216 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dcc1073a0 0x7f5dcc197af0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5dc400c370 tx=0x7f5dc400c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:23.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.201+0000 7f5dc97fa700 1 -- 172.21.15.72:0/4258848216 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc4017070 con 0x7f5dcc1073a0 2024-03-26T02:02:23.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.201+0000 7f5dc97fa700 1 -- 172.21.15.72:0/4258848216 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f5dc400f040 con 0x7f5dcc1073a0 2024-03-26T02:02:23.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.201+0000 7f5dc97fa700 1 -- 172.21.15.72:0/4258848216 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc400e050 con 0x7f5dcc1073a0 2024-03-26T02:02:23.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.201+0000 7f5dd2457700 1 -- 172.21.15.72:0/4258848216 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc00095f0 con 0x7f5dcc1073a0 2024-03-26T02:02:23.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.201+0000 7f5dd2457700 1 -- 172.21.15.72:0/4258848216 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dcc1a4800 con 0x7f5dcc1073a0 2024-03-26T02:02:23.205 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.202+0000 7f5dd2457700 1 -- 172.21.15.72:0/4258848216 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dcc068d40 con 0x7f5dcc1073a0 2024-03-26T02:02:23.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.203+0000 7f5dc97fa700 1 -- 172.21.15.72:0/4258848216 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7f5dc4004470 con 0x7f5dcc1073a0 2024-03-26T02:02:23.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.203+0000 7f5dc97fa700 1 --2- 172.21.15.72:0/4258848216 >> v2:172.21.15.72:6800/3246690770 conn(0x7f5db406da80 0x7f5db406ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:23.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.203+0000 7f5dcbfff700 1 --2- 172.21.15.72:0/4258848216 >> v2:172.21.15.72:6800/3246690770 conn(0x7f5db406da80 0x7f5db406ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:23.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.203+0000 7f5dc97fa700 1 -- 172.21.15.72:0/4258848216 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6259+0+0 (secure 0 0 0) 0x7f5dc408cc50 con 0x7f5dcc1073a0 2024-03-26T02:02:23.206 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.204+0000 7f5dcbfff700 1 --2- 172.21.15.72:0/4258848216 >> v2:172.21.15.72:6800/3246690770 conn(0x7f5db406da80 0x7f5db406ff40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5dc00098e0 tx=0x7f5dc0005960 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:23.208 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.208+0000 7f5dc97fa700 1 -- 172.21.15.72:0/4258848216 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f5dc4092020 con 0x7f5dcc1073a0 2024-03-26T02:02:23.312 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:22 smithi173 ceph-mon[33839]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:23.407 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.407+0000 7f5dd2457700 1 -- 172.21.15.72:0/4258848216 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f5dcc1117a0 con 0x7f5dcc1073a0 2024-03-26T02:02:23.408 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.407+0000 7f5dc97fa700 1 -- 172.21.15.72:0/4258848216 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5dc405a0f0 con 0x7f5dcc1073a0 2024-03-26T02:02:23.408 INFO:teuthology.orchestra.run.smithi072.stdout:55834574899 2024-03-26T02:02:23.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.409+0000 7f5dd2457700 1 -- 172.21.15.72:0/4258848216 >> v2:172.21.15.72:6800/3246690770 conn(0x7f5db406da80 msgr2=0x7f5db406ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:23.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.409+0000 7f5dd2457700 1 --2- 172.21.15.72:0/4258848216 >> v2:172.21.15.72:6800/3246690770 conn(0x7f5db406da80 0x7f5db406ff40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5dc00098e0 tx=0x7f5dc0005960 comp rx=0 tx=0).stop 2024-03-26T02:02:23.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.409+0000 7f5dd2457700 1 -- 172.21.15.72:0/4258848216 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dcc1073a0 msgr2=0x7f5dcc197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:23.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.409+0000 7f5dd2457700 1 --2- 172.21.15.72:0/4258848216 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dcc1073a0 0x7f5dcc197af0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5dc400c370 tx=0x7f5dc400c680 comp rx=0 tx=0).stop 2024-03-26T02:02:23.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.410+0000 7f5dd2457700 1 -- 172.21.15.72:0/4258848216 shutdown_connections 2024-03-26T02:02:23.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.410+0000 7f5dd2457700 1 --2- 172.21.15.72:0/4258848216 >> v2:172.21.15.72:6800/3246690770 conn(0x7f5db406da80 0x7f5db406ff40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:23.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.410+0000 7f5dd2457700 1 --2- 172.21.15.72:0/4258848216 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f5dcc1073a0 0x7f5dcc197af0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:23.410 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.410+0000 7f5dd2457700 1 --2- 172.21.15.72:0/4258848216 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f5dcc1022c0 0x7f5dcc19d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:23.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.410+0000 7f5dd2457700 1 --2- 172.21.15.72:0/4258848216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5dcc102c70 0x7f5dcc19de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:23.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.410+0000 7f5dd2457700 1 -- 172.21.15.72:0/4258848216 >> 172.21.15.72:0/4258848216 conn(0x7f5dcc0fdb30 msgr2=0x7f5dcc0fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:23.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.410+0000 7f5dd2457700 1 -- 172.21.15.72:0/4258848216 shutdown_connections 2024-03-26T02:02:23.411 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:23.410+0000 7f5dd2457700 1 -- 172.21.15.72:0/4258848216 wait complete. 2024-03-26T02:02:23.895 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:23 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4258848216' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T02:02:23.895 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:23 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4258848216' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T02:02:24.177 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574897 got 55834574899 for osd.1 2024-03-26T02:02:24.177 DEBUG:teuthology.parallel:result is None 2024-03-26T02:02:24.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:23 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4258848216' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T02:02:25.168 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:24 smithi072 ceph-mon[41619]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:25.168 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:24 smithi072 ceph-mon[30177]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:25.168 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:02:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:02:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:02:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:24 smithi173 ceph-mon[33839]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:25.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.385+0000 7f60fd1eb700 1 -- 172.21.15.72:0/1215671750 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f60f806a5c0 msgr2=0x7f60f806a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:25.462 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.385+0000 7f60fd1eb700 1 --2- 172.21.15.72:0/1215671750 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f60f806a5c0 0x7f60f806a9a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f60e8009990 tx=0x7f60e8009ca0 comp rx=0 tx=0).stop 2024-03-26T02:02:25.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.385+0000 7f60fd1eb700 1 -- 172.21.15.72:0/1215671750 shutdown_connections 2024-03-26T02:02:25.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.385+0000 7f60fd1eb700 1 --2- 172.21.15.72:0/1215671750 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60f810eb60 0x7f60f8110f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:25.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.385+0000 7f60fd1eb700 1 --2- 172.21.15.72:0/1215671750 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f60f806a5c0 0x7f60f806a9a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:25.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.385+0000 7f60fd1eb700 1 --2- 172.21.15.72:0/1215671750 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60f806aee0 0x7f60f810e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:25.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.385+0000 7f60fd1eb700 1 -- 172.21.15.72:0/1215671750 >> 172.21.15.72:0/1215671750 conn(0x7f60f8076fd0 msgr2=0x7f60f80773e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:25.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.385+0000 7f60fd1eb700 1 -- 172.21.15.72:0/1215671750 shutdown_connections 2024-03-26T02:02:25.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.385+0000 7f60fd1eb700 1 -- 172.21.15.72:0/1215671750 wait complete. 2024-03-26T02:02:25.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.386+0000 7f60fd1eb700 1 Processor -- start 2024-03-26T02:02:25.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.386+0000 7f60fd1eb700 1 -- start start 2024-03-26T02:02:25.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.386+0000 7f60fd1eb700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60f806a5c0 0x7f60f8199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:25.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.386+0000 7f60fd1eb700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f60f806aee0 0x7f60f819a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:25.463 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.386+0000 7f60fd1eb700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60f810eb60 0x7f60f819e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:25.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.386+0000 7f60fd1eb700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f819edd0 con 0x7f60f806a5c0 2024-03-26T02:02:25.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.386+0000 7f60fd1eb700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60f819ef40 con 0x7f60f810eb60 2024-03-26T02:02:25.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.386+0000 7f60fd1eb700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60f819f100 con 0x7f60f806aee0 2024-03-26T02:02:25.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.387+0000 7f60f659c700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f60f806aee0 0x7f60f819a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:25.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.387+0000 7f60f659c700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f60f806aee0 0x7f60f819a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:35206/0 (socket says 172.21.15.72:35206) 2024-03-26T02:02:25.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.387+0000 7f60f659c700 1 -- 172.21.15.72:0/1163296168 learned_addr learned my addr 172.21.15.72:0/1163296168 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:02:25.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.387+0000 7f60f759e700 1 --2- 172.21.15.72:0/1163296168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60f810eb60 0x7f60f819e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:25.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.387+0000 7f60f6d9d700 1 --2- 172.21.15.72:0/1163296168 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60f806a5c0 0x7f60f8199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:25.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.387+0000 7f60f659c700 1 -- 172.21.15.72:0/1163296168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60f810eb60 msgr2=0x7f60f819e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:25.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.387+0000 7f60f659c700 1 --2- 172.21.15.72:0/1163296168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60f810eb60 0x7f60f819e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:25.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.387+0000 7f60f659c700 1 -- 172.21.15.72:0/1163296168 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60f806a5c0 msgr2=0x7f60f8199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:25.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.387+0000 7f60f659c700 1 --2- 172.21.15.72:0/1163296168 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60f806a5c0 0x7f60f8199f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:25.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.387+0000 7f60f659c700 1 -- 172.21.15.72:0/1163296168 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f0009c40 con 0x7f60f806aee0 2024-03-26T02:02:25.464 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.387+0000 7f60f659c700 1 --2- 172.21.15.72:0/1163296168 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f60f806aee0 0x7f60f819a480 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f60ec00d950 tx=0x7f60ec00dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:25.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.387+0000 7f60dffff700 1 -- 172.21.15.72:0/1163296168 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ec009930 con 0x7f60f806aee0 2024-03-26T02:02:25.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.387+0000 7f60fd1eb700 1 -- 172.21.15.72:0/1163296168 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e80095f0 con 0x7f60f806aee0 2024-03-26T02:02:25.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.387+0000 7f60dffff700 1 -- 172.21.15.72:0/1163296168 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f60ec010460 con 0x7f60f806aee0 2024-03-26T02:02:25.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.387+0000 7f60dffff700 1 -- 172.21.15.72:0/1163296168 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ec00f620 con 0x7f60f806aee0 2024-03-26T02:02:25.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.387+0000 7f60fd1eb700 1 -- 172.21.15.72:0/1163296168 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60f81a6a70 con 0x7f60f806aee0 2024-03-26T02:02:25.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.388+0000 7f60fd1eb700 1 -- 172.21.15.72:0/1163296168 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60f8101580 con 0x7f60f806aee0 2024-03-26T02:02:25.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.389+0000 7f60dffff700 1 -- 172.21.15.72:0/1163296168 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7f60ec00f780 con 0x7f60f806aee0 2024-03-26T02:02:25.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.389+0000 7f60dffff700 1 --2- 172.21.15.72:0/1163296168 >> v2:172.21.15.72:6800/3246690770 conn(0x7f60e006da80 0x7f60e006ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:25.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.390+0000 7f60dffff700 1 -- 172.21.15.72:0/1163296168 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6259+0+0 (secure 0 0 0) 0x7f60ec08d1c0 con 0x7f60f806aee0 2024-03-26T02:02:25.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.390+0000 7f60f6d9d700 1 --2- 172.21.15.72:0/1163296168 >> v2:172.21.15.72:6800/3246690770 conn(0x7f60e006da80 0x7f60e006ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:25.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.390+0000 7f60f6d9d700 1 --2- 172.21.15.72:0/1163296168 >> v2:172.21.15.72:6800/3246690770 conn(0x7f60e006da80 0x7f60e006ff40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f60e800b2e0 tx=0x7f60e8005960 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:25.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.394+0000 7f60dffff700 1 -- 172.21.15.72:0/1163296168 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f60ec05a650 con 0x7f60f806aee0 2024-03-26T02:02:25.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.473+0000 7f1c1e41d700 1 -- 172.21.15.72:0/3635147759 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c1806a5c0 msgr2=0x7f1c1806a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:25.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.473+0000 7f1c1e41d700 1 --2- 172.21.15.72:0/3635147759 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c1806a5c0 0x7f1c1806a9a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1c080098f0 tx=0x7f1c08009c00 comp rx=0 tx=0).stop 2024-03-26T02:02:25.563 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.473+0000 7f1c1e41d700 1 -- 172.21.15.72:0/3635147759 shutdown_connections 2024-03-26T02:02:25.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.473+0000 7f1c1e41d700 1 --2- 172.21.15.72:0/3635147759 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c1810eb60 0x7f1c18110f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:25.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.473+0000 7f1c1e41d700 1 --2- 172.21.15.72:0/3635147759 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c1806a5c0 0x7f1c1806a9a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:25.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.473+0000 7f1c1e41d700 1 --2- 172.21.15.72:0/3635147759 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1c1806aee0 0x7f1c1810e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:25.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.473+0000 7f1c1e41d700 1 -- 172.21.15.72:0/3635147759 >> 172.21.15.72:0/3635147759 conn(0x7f1c18076fd0 msgr2=0x7f1c180773e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:25.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.473+0000 7f1c1e41d700 1 -- 172.21.15.72:0/3635147759 shutdown_connections 2024-03-26T02:02:25.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.473+0000 7f1c1e41d700 1 -- 172.21.15.72:0/3635147759 wait complete. 2024-03-26T02:02:25.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.474+0000 7f1c1e41d700 1 Processor -- start 2024-03-26T02:02:25.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.474+0000 7f1c1e41d700 1 -- start start 2024-03-26T02:02:25.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.474+0000 7f1c1e41d700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c1806a5c0 0x7f1c18199f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:25.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.474+0000 7f1c1e41d700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c1806aee0 0x7f1c1819a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:25.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.474+0000 7f1c1e41d700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1c1810eb60 0x7f1c1819e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:25.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.474+0000 7f1c1e41d700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1819ed90 con 0x7f1c1806aee0 2024-03-26T02:02:25.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.474+0000 7f1c1e41d700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1819ef00 con 0x7f1c1806a5c0 2024-03-26T02:02:25.564 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.474+0000 7f1c1e41d700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c1819f0c0 con 0x7f1c1810eb60 2024-03-26T02:02:25.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.474+0000 7f1c177fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c1806aee0 0x7f1c1819a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:25.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.474+0000 7f1c177fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c1806aee0 0x7f1c1819a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40112/0 (socket says 172.21.15.72:40112) 2024-03-26T02:02:25.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.474+0000 7f1c177fe700 1 -- 172.21.15.72:0/856700018 learned_addr learned my addr 172.21.15.72:0/856700018 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:02:25.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.475+0000 7f1c1c9ba700 1 --2- 172.21.15.72:0/856700018 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1c1810eb60 0x7f1c1819e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:25.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.475+0000 7f1c177fe700 1 -- 172.21.15.72:0/856700018 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1c1810eb60 msgr2=0x7f1c1819e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:25.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.475+0000 7f1c17fff700 1 --2- 172.21.15.72:0/856700018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c1806a5c0 0x7f1c18199f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:25.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.475+0000 7f1c177fe700 1 --2- 172.21.15.72:0/856700018 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1c1810eb60 0x7f1c1819e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:25.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.475+0000 7f1c177fe700 1 -- 172.21.15.72:0/856700018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c1806a5c0 msgr2=0x7f1c18199f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:25.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.475+0000 7f1c177fe700 1 --2- 172.21.15.72:0/856700018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c1806a5c0 0x7f1c18199f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:25.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.475+0000 7f1c177fe700 1 -- 172.21.15.72:0/856700018 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0c0095f0 con 0x7f1c1806aee0 2024-03-26T02:02:25.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.475+0000 7f1c177fe700 1 --2- 172.21.15.72:0/856700018 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c1806aee0 0x7f1c1819a440 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c00ec10 tx=0x7f1c0c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:25.565 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.475+0000 7f1c157fa700 1 -- 172.21.15.72:0/856700018 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c00cd40 con 0x7f1c1806aee0 2024-03-26T02:02:25.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.475+0000 7f1c1e41d700 1 -- 172.21.15.72:0/856700018 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c08009520 con 0x7f1c1806aee0 2024-03-26T02:02:25.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.475+0000 7f1c157fa700 1 -- 172.21.15.72:0/856700018 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f1c0c0044d0 con 0x7f1c1806aee0 2024-03-26T02:02:25.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.475+0000 7f1c157fa700 1 -- 172.21.15.72:0/856700018 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c010430 con 0x7f1c1806aee0 2024-03-26T02:02:25.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.475+0000 7f1c1e41d700 1 -- 172.21.15.72:0/856700018 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c181a6a30 con 0x7f1c1806aee0 2024-03-26T02:02:25.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.476+0000 7f1c1e41d700 1 -- 172.21.15.72:0/856700018 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c18068d40 con 0x7f1c1806aee0 2024-03-26T02:02:25.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.477+0000 7f1c157fa700 1 -- 172.21.15.72:0/856700018 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7f1c0c010630 con 0x7f1c1806aee0 2024-03-26T02:02:25.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.477+0000 7f1c157fa700 1 --2- 172.21.15.72:0/856700018 >> v2:172.21.15.72:6800/3246690770 conn(0x7f1c0006da80 0x7f1c0006ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:25.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.477+0000 7f1c157fa700 1 -- 172.21.15.72:0/856700018 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6259+0+0 (secure 0 0 0) 0x7f1c0c014070 con 0x7f1c1806aee0 2024-03-26T02:02:25.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.477+0000 7f1c17fff700 1 --2- 172.21.15.72:0/856700018 >> v2:172.21.15.72:6800/3246690770 conn(0x7f1c0006da80 0x7f1c0006ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:25.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.478+0000 7f1c17fff700 1 --2- 172.21.15.72:0/856700018 >> v2:172.21.15.72:6800/3246690770 conn(0x7f1c0006da80 0x7f1c0006ff40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1c08009230 tx=0x7f1c08005b40 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:25.566 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.482+0000 7f1c157fa700 1 -- 172.21.15.72:0/856700018 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f1c0c05ae40 con 0x7f1c1806aee0 2024-03-26T02:02:25.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.628+0000 7f60fd1eb700 1 -- 172.21.15.72:0/1163296168 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f60f819b250 con 0x7f60f806aee0 2024-03-26T02:02:25.629 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.628+0000 7f60dffff700 1 -- 172.21.15.72:0/1163296168 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f60ec05a1e0 con 0x7f60f806aee0 2024-03-26T02:02:25.629 INFO:teuthology.orchestra.run.smithi072.stdout:163208757277 2024-03-26T02:02:25.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.631+0000 7f60fd1eb700 1 -- 172.21.15.72:0/1163296168 >> v2:172.21.15.72:6800/3246690770 conn(0x7f60e006da80 msgr2=0x7f60e006ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:25.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.631+0000 7f60fd1eb700 1 --2- 172.21.15.72:0/1163296168 >> v2:172.21.15.72:6800/3246690770 conn(0x7f60e006da80 0x7f60e006ff40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f60e800b2e0 tx=0x7f60e8005960 comp rx=0 tx=0).stop 2024-03-26T02:02:25.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.631+0000 7f60fd1eb700 1 -- 172.21.15.72:0/1163296168 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f60f806aee0 msgr2=0x7f60f819a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:25.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.631+0000 7f60fd1eb700 1 --2- 172.21.15.72:0/1163296168 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f60f806aee0 0x7f60f819a480 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f60ec00d950 tx=0x7f60ec00dc60 comp rx=0 tx=0).stop 2024-03-26T02:02:25.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.631+0000 7f60fd1eb700 1 -- 172.21.15.72:0/1163296168 shutdown_connections 2024-03-26T02:02:25.631 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.631+0000 7f60fd1eb700 1 --2- 172.21.15.72:0/1163296168 >> v2:172.21.15.72:6800/3246690770 conn(0x7f60e006da80 0x7f60e006ff40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:25.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.631+0000 7f60fd1eb700 1 --2- 172.21.15.72:0/1163296168 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60f810eb60 0x7f60f819e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:25.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.631+0000 7f60fd1eb700 1 --2- 172.21.15.72:0/1163296168 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f60f806a5c0 0x7f60f8199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:25.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.631+0000 7f60fd1eb700 1 --2- 172.21.15.72:0/1163296168 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f60f806aee0 0x7f60f819a480 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:25.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.631+0000 7f60fd1eb700 1 -- 172.21.15.72:0/1163296168 >> 172.21.15.72:0/1163296168 conn(0x7f60f8076fd0 msgr2=0x7f60f80fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:25.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.631+0000 7f60fd1eb700 1 -- 172.21.15.72:0/1163296168 shutdown_connections 2024-03-26T02:02:25.632 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.631+0000 7f60fd1eb700 1 -- 172.21.15.72:0/1163296168 wait complete. 2024-03-26T02:02:25.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.703+0000 7f1c1e41d700 1 -- 172.21.15.72:0/856700018 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f1c18113960 con 0x7f1c1806aee0 2024-03-26T02:02:25.704 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.704+0000 7f1c157fa700 1 -- 172.21.15.72:0/856700018 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1c0c05a9d0 con 0x7f1c1806aee0 2024-03-26T02:02:25.705 INFO:teuthology.orchestra.run.smithi072.stdout:34359738424 2024-03-26T02:02:25.706 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.706+0000 7f1c1e41d700 1 -- 172.21.15.72:0/856700018 >> v2:172.21.15.72:6800/3246690770 conn(0x7f1c0006da80 msgr2=0x7f1c0006ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:25.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.706+0000 7f1c1e41d700 1 --2- 172.21.15.72:0/856700018 >> v2:172.21.15.72:6800/3246690770 conn(0x7f1c0006da80 0x7f1c0006ff40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1c08009230 tx=0x7f1c08005b40 comp rx=0 tx=0).stop 2024-03-26T02:02:25.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.706+0000 7f1c1e41d700 1 -- 172.21.15.72:0/856700018 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c1806aee0 msgr2=0x7f1c1819a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:25.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.706+0000 7f1c1e41d700 1 --2- 172.21.15.72:0/856700018 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c1806aee0 0x7f1c1819a440 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c00ec10 tx=0x7f1c0c00ef20 comp rx=0 tx=0).stop 2024-03-26T02:02:25.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.706+0000 7f1c1e41d700 1 -- 172.21.15.72:0/856700018 shutdown_connections 2024-03-26T02:02:25.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.706+0000 7f1c1e41d700 1 --2- 172.21.15.72:0/856700018 >> v2:172.21.15.72:6800/3246690770 conn(0x7f1c0006da80 0x7f1c0006ff40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:25.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.706+0000 7f1c1e41d700 1 --2- 172.21.15.72:0/856700018 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1c1810eb60 0x7f1c1819e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:25.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.706+0000 7f1c1e41d700 1 --2- 172.21.15.72:0/856700018 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1c1806a5c0 0x7f1c18199f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:25.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.706+0000 7f1c1e41d700 1 --2- 172.21.15.72:0/856700018 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1c1806aee0 0x7f1c1819a440 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:25.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.706+0000 7f1c1e41d700 1 -- 172.21.15.72:0/856700018 >> 172.21.15.72:0/856700018 conn(0x7f1c18076fd0 msgr2=0x7f1c180fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:25.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.706+0000 7f1c1e41d700 1 -- 172.21.15.72:0/856700018 shutdown_connections 2024-03-26T02:02:25.707 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:25.706+0000 7f1c1e41d700 1 -- 172.21.15.72:0/856700018 wait complete. 2024-03-26T02:02:26.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:26 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:26.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:26 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:26.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:26 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1163296168' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T02:02:26.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:26 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/856700018' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T02:02:26.449 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:26 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:26.450 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:26 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:26.450 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:26 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1163296168' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T02:02:26.450 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:26 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/856700018' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T02:02:26.450 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:26 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:26.450 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:26 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:26.450 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:26 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1163296168' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T02:02:26.450 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:26 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/856700018' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T02:02:26.475 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738419 got 34359738424 for osd.0 2024-03-26T02:02:26.475 DEBUG:teuthology.parallel:result is None 2024-03-26T02:02:26.909 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757272 got 163208757277 for osd.5 2024-03-26T02:02:26.909 DEBUG:teuthology.parallel:result is None 2024-03-26T02:02:26.909 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-26T02:02:26.909 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T02:02:27.165 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:02:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:27 smithi072 ceph-mon[30177]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:27 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:27 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:27 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:02:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:27 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:02:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:27 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:02:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:27 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:02:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:27 smithi072 ceph-mon[41619]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:27 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:27 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:27 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:02:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:27 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:02:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:27 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:02:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:27 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:02:27.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:27 smithi173 ceph-mon[33839]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:27.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:27 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:27.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:27 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:27.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:27 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:02:27.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:27 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:02:27.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:27 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:02:27.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:27 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:02:28.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:28 smithi173 ceph-mon[33839]: Adjusting osd_memory_target on smithi072 to 3760M 2024-03-26T02:02:28.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:28 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:28.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:28 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:02:28.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:28 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:02:28.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:28 smithi173 ceph-mon[33839]: Updating smithi072:/etc/ceph/ceph.conf 2024-03-26T02:02:28.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:28 smithi173 ceph-mon[33839]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-26T02:02:28.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:28 smithi072 ceph-mon[41619]: Adjusting osd_memory_target on smithi072 to 3760M 2024-03-26T02:02:28.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:28 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:28.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:28 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:02:28.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:28 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:02:28.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:28 smithi072 ceph-mon[41619]: Updating smithi072:/etc/ceph/ceph.conf 2024-03-26T02:02:28.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:28 smithi072 ceph-mon[41619]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-26T02:02:28.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:28 smithi072 ceph-mon[30177]: Adjusting osd_memory_target on smithi072 to 3760M 2024-03-26T02:02:28.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:28 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:28.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:28 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:02:28.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:28 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:02:28.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:28 smithi072 ceph-mon[30177]: Updating smithi072:/etc/ceph/ceph.conf 2024-03-26T02:02:28.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:28 smithi072 ceph-mon[30177]: Updating smithi173:/etc/ceph/ceph.conf 2024-03-26T02:02:29.014 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.013+0000 7f30846e2700 1 -- 172.21.15.72:0/18760779 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f307c1020d0 msgr2=0x7f307c102550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:29.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.013+0000 7f30846e2700 1 --2- 172.21.15.72:0/18760779 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f307c1020d0 0x7f307c102550 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f306c0099c0 tx=0x7f306c009cd0 comp rx=0 tx=0).stop 2024-03-26T02:02:29.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.014+0000 7f30846e2700 1 -- 172.21.15.72:0/18760779 shutdown_connections 2024-03-26T02:02:29.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.014+0000 7f30846e2700 1 --2- 172.21.15.72:0/18760779 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f307c102a90 0x7f307c10eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:29.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.014+0000 7f30846e2700 1 --2- 172.21.15.72:0/18760779 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f307c1080b0 0x7f307c108490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:29.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.014+0000 7f30846e2700 1 --2- 172.21.15.72:0/18760779 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f307c1020d0 0x7f307c102550 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:29.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.014+0000 7f30846e2700 1 -- 172.21.15.72:0/18760779 >> 172.21.15.72:0/18760779 conn(0x7f307c0791b0 msgr2=0x7f307c0795c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:29.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.014+0000 7f30846e2700 1 -- 172.21.15.72:0/18760779 shutdown_connections 2024-03-26T02:02:29.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.014+0000 7f30846e2700 1 -- 172.21.15.72:0/18760779 wait complete. 2024-03-26T02:02:29.015 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.014+0000 7f30846e2700 1 Processor -- start 2024-03-26T02:02:29.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.014+0000 7f30846e2700 1 -- start start 2024-03-26T02:02:29.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.015+0000 7f30846e2700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f307c1020d0 0x7f307c19d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:29.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.015+0000 7f30846e2700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f307c102a90 0x7f307c19de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:29.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.015+0000 7f30846e2700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f307c1080b0 0x7f307c197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:29.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.015+0000 7f30846e2700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c1980e0 con 0x7f307c1020d0 2024-03-26T02:02:29.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.015+0000 7f30846e2700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c198220 con 0x7f307c102a90 2024-03-26T02:02:29.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.015+0000 7f30846e2700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307c1983e0 con 0x7f307c1080b0 2024-03-26T02:02:29.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.015+0000 7f308247e700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f307c1020d0 0x7f307c19d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:29.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.015+0000 7f308247e700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f307c1020d0 0x7f307c19d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40142/0 (socket says 172.21.15.72:40142) 2024-03-26T02:02:29.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.015+0000 7f308247e700 1 -- 172.21.15.72:0/3662145723 learned_addr learned my addr 172.21.15.72:0/3662145723 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:02:29.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.015+0000 7f3082c7f700 1 --2- 172.21.15.72:0/3662145723 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f307c1080b0 0x7f307c197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:29.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.015+0000 7f3081c7d700 1 --2- 172.21.15.72:0/3662145723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f307c102a90 0x7f307c19de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:29.017 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.015+0000 7f308247e700 1 -- 172.21.15.72:0/3662145723 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f307c1080b0 msgr2=0x7f307c197a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:29.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.015+0000 7f308247e700 1 --2- 172.21.15.72:0/3662145723 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f307c1080b0 0x7f307c197a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:29.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.015+0000 7f308247e700 1 -- 172.21.15.72:0/3662145723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f307c102a90 msgr2=0x7f307c19de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:29.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.015+0000 7f308247e700 1 --2- 172.21.15.72:0/3662145723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f307c102a90 0x7f307c19de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:29.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.015+0000 7f308247e700 1 -- 172.21.15.72:0/3662145723 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3078009520 con 0x7f307c1020d0 2024-03-26T02:02:29.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.015+0000 7f3081c7d700 1 --2- 172.21.15.72:0/3662145723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f307c102a90 0x7f307c19de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T02:02:29.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.015+0000 7f308247e700 1 --2- 172.21.15.72:0/3662145723 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f307c1020d0 0x7f307c19d8c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f307800eb30 tx=0x7f307800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:29.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.015+0000 7f30737fe700 1 -- 172.21.15.72:0/3662145723 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3078009880 con 0x7f307c1020d0 2024-03-26T02:02:29.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.016+0000 7f30737fe700 1 -- 172.21.15.72:0/3662145723 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f30780044d0 con 0x7f307c1020d0 2024-03-26T02:02:29.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.016+0000 7f30737fe700 1 -- 172.21.15.72:0/3662145723 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3078003e20 con 0x7f307c1020d0 2024-03-26T02:02:29.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.016+0000 7f30846e2700 1 -- 172.21.15.72:0/3662145723 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f306c0095f0 con 0x7f307c1020d0 2024-03-26T02:02:29.018 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.016+0000 7f30846e2700 1 -- 172.21.15.72:0/3662145723 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f307c06bb20 con 0x7f307c1020d0 2024-03-26T02:02:29.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.017+0000 7f30846e2700 1 -- 172.21.15.72:0/3662145723 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f307c103760 con 0x7f307c1020d0 2024-03-26T02:02:29.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.017+0000 7f30737fe700 1 -- 172.21.15.72:0/3662145723 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7f3078003f80 con 0x7f307c1020d0 2024-03-26T02:02:29.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.018+0000 7f30737fe700 1 --2- 172.21.15.72:0/3662145723 >> v2:172.21.15.72:6800/3246690770 conn(0x7f306806da80 0x7f306806ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:29.019 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.018+0000 7f3081c7d700 1 --2- 172.21.15.72:0/3662145723 >> v2:172.21.15.72:6800/3246690770 conn(0x7f306806da80 0x7f306806ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:29.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.018+0000 7f30737fe700 1 -- 172.21.15.72:0/3662145723 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6259+0+0 (secure 0 0 0) 0x7f3078014070 con 0x7f307c1020d0 2024-03-26T02:02:29.020 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.018+0000 7f3081c7d700 1 --2- 172.21.15.72:0/3662145723 >> v2:172.21.15.72:6800/3246690770 conn(0x7f306806da80 0x7f306806ff40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f306c000c00 tx=0x7f306c005ab0 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:29.024 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.024+0000 7f30737fe700 1 -- 172.21.15.72:0/3662145723 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f307805ab90 con 0x7f307c1020d0 2024-03-26T02:02:29.224 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.224+0000 7f30846e2700 1 -- 172.21.15.72:0/3662145723 --> v2:172.21.15.72:6800/3246690770 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f307c198fb0 con 0x7f306806da80 2024-03-26T02:02:29.234 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.234+0000 7f30737fe700 1 -- 172.21.15.72:0/3662145723 <== mgr.24467 v2:172.21.15.72:6800/3246690770 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+346503 (secure 0 0 0) 0x7f307c198fb0 con 0x7f306806da80 2024-03-26T02:02:29.239 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T02:02:29.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.237+0000 7f30846e2700 1 -- 172.21.15.72:0/3662145723 >> v2:172.21.15.72:6800/3246690770 conn(0x7f306806da80 msgr2=0x7f306806ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:29.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.237+0000 7f30846e2700 1 --2- 172.21.15.72:0/3662145723 >> v2:172.21.15.72:6800/3246690770 conn(0x7f306806da80 0x7f306806ff40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f306c000c00 tx=0x7f306c005ab0 comp rx=0 tx=0).stop 2024-03-26T02:02:29.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.237+0000 7f30846e2700 1 -- 172.21.15.72:0/3662145723 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f307c1020d0 msgr2=0x7f307c19d8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:29.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.237+0000 7f30846e2700 1 --2- 172.21.15.72:0/3662145723 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f307c1020d0 0x7f307c19d8c0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f307800eb30 tx=0x7f307800ee40 comp rx=0 tx=0).stop 2024-03-26T02:02:29.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.238+0000 7f30846e2700 1 -- 172.21.15.72:0/3662145723 shutdown_connections 2024-03-26T02:02:29.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.238+0000 7f30846e2700 1 --2- 172.21.15.72:0/3662145723 >> v2:172.21.15.72:6800/3246690770 conn(0x7f306806da80 0x7f306806ff40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:29.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.238+0000 7f30846e2700 1 --2- 172.21.15.72:0/3662145723 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f307c1080b0 0x7f307c197a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:29.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.238+0000 7f30846e2700 1 --2- 172.21.15.72:0/3662145723 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f307c1020d0 0x7f307c19d8c0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:29.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.238+0000 7f30846e2700 1 --2- 172.21.15.72:0/3662145723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f307c102a90 0x7f307c19de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:29.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.238+0000 7f30846e2700 1 -- 172.21.15.72:0/3662145723 >> 172.21.15.72:0/3662145723 conn(0x7f307c0791b0 msgr2=0x7f307c105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:29.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.238+0000 7f30846e2700 1 -- 172.21.15.72:0/3662145723 shutdown_connections 2024-03-26T02:02:29.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:29.238+0000 7f30846e2700 1 -- 172.21.15.72:0/3662145723 wait complete. 2024-03-26T02:02:29.243 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-26T02:02:29.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:29 smithi072 ceph-mon[30177]: Updating smithi072:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:02:29.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:29 smithi072 ceph-mon[30177]: Updating smithi173:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:02:29.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:29 smithi072 ceph-mon[30177]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:29.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:29 smithi072 ceph-mon[30177]: Updating smithi072:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:02:29.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:29 smithi072 ceph-mon[30177]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:02:29.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:29 smithi072 ceph-mon[30177]: Updating smithi072:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:02:29.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:29 smithi072 ceph-mon[30177]: Updating smithi173:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:02:29.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:29 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:29.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:29 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:29.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:29 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:29.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:29 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:29.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:29 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:29.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:29 smithi072 ceph-mon[30177]: Deploying daemon alertmanager.a on smithi072 2024-03-26T02:02:29.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:29 smithi072 ceph-mon[41619]: Updating smithi072:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:02:29.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:29 smithi072 ceph-mon[41619]: Updating smithi173:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:02:29.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:29 smithi072 ceph-mon[41619]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:29.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:29 smithi072 ceph-mon[41619]: Updating smithi072:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:02:29.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:29 smithi072 ceph-mon[41619]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:02:29.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:29 smithi072 ceph-mon[41619]: Updating smithi072:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:02:29.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:29 smithi072 ceph-mon[41619]: Updating smithi173:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:02:29.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:29 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:29.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:29 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:29.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:29 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:29.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:29 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:29.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:29 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:29.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:29 smithi072 ceph-mon[41619]: Deploying daemon alertmanager.a on smithi072 2024-03-26T02:02:29.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:29 smithi173 ceph-mon[33839]: Updating smithi072:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:02:29.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:29 smithi173 ceph-mon[33839]: Updating smithi173:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T02:02:29.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:29 smithi173 ceph-mon[33839]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:29.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:29 smithi173 ceph-mon[33839]: Updating smithi072:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:02:29.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:29 smithi173 ceph-mon[33839]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-03-26T02:02:29.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:29 smithi173 ceph-mon[33839]: Updating smithi072:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:02:29.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:29 smithi173 ceph-mon[33839]: Updating smithi173:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T02:02:29.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:29 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:29.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:29 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:29.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:29 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:29.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:29 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:29.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:29 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:29.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:29 smithi173 ceph-mon[33839]: Deploying daemon alertmanager.a on smithi072 2024-03-26T02:02:30.249 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-03-26T02:02:27.437535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1293,"num_read_kb":1107,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":797,"ondisk_log_size":797,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240088,"kb_used_data":7292,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749721128,"statfs":{"total":767960285184,"available":767714435072,"internally_reserved":0,"allocated":7467008,"data_stored":3990386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003090"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419453+0000","last_change":"2024-03-26T02:01:14.254100+0000","last_active":"2024-03-26T02:01:49.419453+0000","last_peered":"2024-03-26T02:01:49.419453+0000","last_clean":"2024-03-26T02:01:49.419453+0000","last_became_active":"2024-03-26T02:01:14.253800+0000","last_became_peered":"2024-03-26T02:01:14.253800+0000","last_unstale":"2024-03-26T02:01:49.419453+0000","last_undegraded":"2024-03-26T02:01:49.419453+0000","last_fullsized":"2024-03-26T02:01:49.419453+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:32:05.985959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.738640+0000","last_change":"2024-03-26T02:01:12.244272+0000","last_active":"2024-03-26T02:01:49.738640+0000","last_peered":"2024-03-26T02:01:49.738640+0000","last_clean":"2024-03-26T02:01:49.738640+0000","last_became_active":"2024-03-26T02:01:12.243977+0000","last_became_peered":"2024-03-26T02:01:12.243977+0000","last_unstale":"2024-03-26T02:01:49.738640+0000","last_undegraded":"2024-03-26T02:01:49.738640+0000","last_fullsized":"2024-03-26T02:01:49.738640+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:47:52.673808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739283+0000","last_change":"2024-03-26T02:01:08.213024+0000","last_active":"2024-03-26T02:01:49.739283+0000","last_peered":"2024-03-26T02:01:49.739283+0000","last_clean":"2024-03-26T02:01:49.739283+0000","last_became_active":"2024-03-26T02:01:08.212743+0000","last_became_peered":"2024-03-26T02:01:08.212743+0000","last_unstale":"2024-03-26T02:01:49.739283+0000","last_undegraded":"2024-03-26T02:01:49.739283+0000","last_fullsized":"2024-03-26T02:01:49.739283+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:48:36.119186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'15","reported_seq":57,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419365+0000","last_change":"2024-03-26T02:01:10.224453+0000","last_active":"2024-03-26T02:01:49.419365+0000","last_peered":"2024-03-26T02:01:49.419365+0000","last_clean":"2024-03-26T02:01:49.419365+0000","last_became_active":"2024-03-26T02:01:10.223780+0000","last_became_peered":"2024-03-26T02:01:10.223780+0000","last_unstale":"2024-03-26T02:01:49.419365+0000","last_undegraded":"2024-03-26T02:01:49.419365+0000","last_fullsized":"2024-03-26T02:01:49.419365+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:08:07.751617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419502+0000","last_change":"2024-03-26T02:01:14.949106+0000","last_active":"2024-03-26T02:01:49.419502+0000","last_peered":"2024-03-26T02:01:49.419502+0000","last_clean":"2024-03-26T02:01:49.419502+0000","last_became_active":"2024-03-26T02:01:14.948745+0000","last_became_peered":"2024-03-26T02:01:14.948745+0000","last_unstale":"2024-03-26T02:01:49.419502+0000","last_undegraded":"2024-03-26T02:01:49.419502+0000","last_fullsized":"2024-03-26T02:01:49.419502+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:50:07.943908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419368+0000","last_change":"2024-03-26T02:01:12.241704+0000","last_active":"2024-03-26T02:01:49.419368+0000","last_peered":"2024-03-26T02:01:49.419368+0000","last_clean":"2024-03-26T02:01:49.419368+0000","last_became_active":"2024-03-26T02:01:12.240025+0000","last_became_peered":"2024-03-26T02:01:12.240025+0000","last_unstale":"2024-03-26T02:01:49.419368+0000","last_undegraded":"2024-03-26T02:01:49.419368+0000","last_fullsized":"2024-03-26T02:01:49.419368+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:37:28.701177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419631+0000","last_change":"2024-03-26T02:01:08.206463+0000","last_active":"2024-03-26T02:01:49.419631+0000","last_peered":"2024-03-26T02:01:49.419631+0000","last_clean":"2024-03-26T02:01:49.419631+0000","last_became_active":"2024-03-26T02:01:08.205735+0000","last_became_peered":"2024-03-26T02:01:08.205735+0000","last_unstale":"2024-03-26T02:01:49.419631+0000","last_undegraded":"2024-03-26T02:01:49.419631+0000","last_fullsized":"2024-03-26T02:01:49.419631+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:04:31.024748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'39","reported_seq":102,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.418742+0000","last_change":"2024-03-26T02:01:10.220412+0000","last_active":"2024-03-26T02:01:49.418742+0000","last_peered":"2024-03-26T02:01:49.418742+0000","last_clean":"2024-03-26T02:01:49.418742+0000","last_became_active":"2024-03-26T02:01:10.220109+0000","last_became_peered":"2024-03-26T02:01:10.220109+0000","last_unstale":"2024-03-26T02:01:49.418742+0000","last_undegraded":"2024-03-26T02:01:49.418742+0000","last_fullsized":"2024-03-26T02:01:49.418742+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:53:46.904838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.737817+0000","last_change":"2024-03-26T02:01:14.250830+0000","last_active":"2024-03-26T02:01:49.737817+0000","last_peered":"2024-03-26T02:01:49.737817+0000","last_clean":"2024-03-26T02:01:49.737817+0000","last_became_active":"2024-03-26T02:01:14.250492+0000","last_became_peered":"2024-03-26T02:01:14.250492+0000","last_unstale":"2024-03-26T02:01:49.737817+0000","last_undegraded":"2024-03-26T02:01:49.737817+0000","last_fullsized":"2024-03-26T02:01:49.737817+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:33:15.249424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.533755+0000","last_change":"2024-03-26T02:01:12.242865+0000","last_active":"2024-03-26T02:01:49.533755+0000","last_peered":"2024-03-26T02:01:49.533755+0000","last_clean":"2024-03-26T02:01:49.533755+0000","last_became_active":"2024-03-26T02:01:12.242552+0000","last_became_peered":"2024-03-26T02:01:12.242552+0000","last_unstale":"2024-03-26T02:01:49.533755+0000","last_undegraded":"2024-03-26T02:01:49.533755+0000","last_fullsized":"2024-03-26T02:01:49.533755+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:07:43.897667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.533976+0000","last_change":"2024-03-26T02:01:08.214349+0000","last_active":"2024-03-26T02:01:49.533976+0000","last_peered":"2024-03-26T02:01:49.533976+0000","last_clean":"2024-03-26T02:01:49.533976+0000","last_became_active":"2024-03-26T02:01:08.214041+0000","last_became_peered":"2024-03-26T02:01:08.214041+0000","last_unstale":"2024-03-26T02:01:49.533976+0000","last_undegraded":"2024-03-26T02:01:49.533976+0000","last_fullsized":"2024-03-26T02:01:49.533976+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:25:28.902727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'21","reported_seq":69,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419730+0000","last_change":"2024-03-26T02:01:10.222145+0000","last_active":"2024-03-26T02:01:49.419730+0000","last_peered":"2024-03-26T02:01:49.419730+0000","last_clean":"2024-03-26T02:01:49.419730+0000","last_became_active":"2024-03-26T02:01:10.221725+0000","last_became_peered":"2024-03-26T02:01:10.221725+0000","last_unstale":"2024-03-26T02:01:49.419730+0000","last_undegraded":"2024-03-26T02:01:49.419730+0000","last_fullsized":"2024-03-26T02:01:49.419730+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:30:14.673620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.533437+0000","last_change":"2024-03-26T02:01:14.254015+0000","last_active":"2024-03-26T02:01:49.533437+0000","last_peered":"2024-03-26T02:01:49.533437+0000","last_clean":"2024-03-26T02:01:49.533437+0000","last_became_active":"2024-03-26T02:01:14.253698+0000","last_became_peered":"2024-03-26T02:01:14.253698+0000","last_unstale":"2024-03-26T02:01:49.533437+0000","last_undegraded":"2024-03-26T02:01:49.533437+0000","last_fullsized":"2024-03-26T02:01:49.533437+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:27:07.758079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.427713+0000","last_change":"2024-03-26T02:01:12.247571+0000","last_active":"2024-03-26T02:01:49.427713+0000","last_peered":"2024-03-26T02:01:49.427713+0000","last_clean":"2024-03-26T02:01:49.427713+0000","last_became_active":"2024-03-26T02:01:12.247293+0000","last_became_peered":"2024-03-26T02:01:12.247293+0000","last_unstale":"2024-03-26T02:01:49.427713+0000","last_undegraded":"2024-03-26T02:01:49.427713+0000","last_fullsized":"2024-03-26T02:01:49.427713+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:36:47.223790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.533888+0000","last_change":"2024-03-26T02:01:08.211815+0000","last_active":"2024-03-26T02:01:49.533888+0000","last_peered":"2024-03-26T02:01:49.533888+0000","last_clean":"2024-03-26T02:01:49.533888+0000","last_became_active":"2024-03-26T02:01:08.211498+0000","last_became_peered":"2024-03-26T02:01:08.211498+0000","last_unstale":"2024-03-26T02:01:49.533888+0000","last_undegraded":"2024-03-26T02:01:49.533888+0000","last_fullsized":"2024-03-26T02:01:49.533888+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:41:29.582259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'11","reported_seq":51,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419974+0000","last_change":"2024-03-26T02:01:10.227169+0000","last_active":"2024-03-26T02:01:49.419974+0000","last_peered":"2024-03-26T02:01:49.419974+0000","last_clean":"2024-03-26T02:01:49.419974+0000","last_became_active":"2024-03-26T02:01:10.226812+0000","last_became_peered":"2024-03-26T02:01:10.226812+0000","last_unstale":"2024-03-26T02:01:49.419974+0000","last_undegraded":"2024-03-26T02:01:49.419974+0000","last_fullsized":"2024-03-26T02:01:49.419974+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:46:25.654115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.420522+0000","last_change":"2024-03-26T02:01:14.947318+0000","last_active":"2024-03-26T02:01:49.420522+0000","last_peered":"2024-03-26T02:01:49.420522+0000","last_clean":"2024-03-26T02:01:49.420522+0000","last_became_active":"2024-03-26T02:01:14.947010+0000","last_became_peered":"2024-03-26T02:01:14.947010+0000","last_unstale":"2024-03-26T02:01:49.420522+0000","last_undegraded":"2024-03-26T02:01:49.420522+0000","last_fullsized":"2024-03-26T02:01:49.420522+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:21:01.716389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.738374+0000","last_change":"2024-03-26T02:01:12.243576+0000","last_active":"2024-03-26T02:01:49.738374+0000","last_peered":"2024-03-26T02:01:49.738374+0000","last_clean":"2024-03-26T02:01:49.738374+0000","last_became_active":"2024-03-26T02:01:12.243288+0000","last_became_peered":"2024-03-26T02:01:12.243288+0000","last_unstale":"2024-03-26T02:01:49.738374+0000","last_undegraded":"2024-03-26T02:01:49.738374+0000","last_fullsized":"2024-03-26T02:01:49.738374+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:57:53.502381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739534+0000","last_change":"2024-03-26T02:01:08.212432+0000","last_active":"2024-03-26T02:01:49.739534+0000","last_peered":"2024-03-26T02:01:49.739534+0000","last_clean":"2024-03-26T02:01:49.739534+0000","last_became_active":"2024-03-26T02:01:08.211840+0000","last_became_peered":"2024-03-26T02:01:08.211840+0000","last_unstale":"2024-03-26T02:01:49.739534+0000","last_undegraded":"2024-03-26T02:01:49.739534+0000","last_fullsized":"2024-03-26T02:01:49.739534+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:22:47.051270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.422035+0000","last_change":"2024-03-26T02:01:10.223652+0000","last_active":"2024-03-26T02:01:49.422035+0000","last_peered":"2024-03-26T02:01:49.422035+0000","last_clean":"2024-03-26T02:01:49.422035+0000","last_became_active":"2024-03-26T02:01:10.223283+0000","last_became_peered":"2024-03-26T02:01:10.223283+0000","last_unstale":"2024-03-26T02:01:49.422035+0000","last_undegraded":"2024-03-26T02:01:49.422035+0000","last_fullsized":"2024-03-26T02:01:49.422035+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:15:56.314001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419792+0000","last_change":"2024-03-26T02:01:14.947449+0000","last_active":"2024-03-26T02:01:49.419792+0000","last_peered":"2024-03-26T02:01:49.419792+0000","last_clean":"2024-03-26T02:01:49.419792+0000","last_became_active":"2024-03-26T02:01:14.947074+0000","last_became_peered":"2024-03-26T02:01:14.947074+0000","last_unstale":"2024-03-26T02:01:49.419792+0000","last_undegraded":"2024-03-26T02:01:49.419792+0000","last_fullsized":"2024-03-26T02:01:49.419792+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:05:57.273489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.420106+0000","last_change":"2024-03-26T02:01:12.237378+0000","last_active":"2024-03-26T02:01:49.420106+0000","last_peered":"2024-03-26T02:01:49.420106+0000","last_clean":"2024-03-26T02:01:49.420106+0000","last_became_active":"2024-03-26T02:01:12.237085+0000","last_became_peered":"2024-03-26T02:01:12.237085+0000","last_unstale":"2024-03-26T02:01:49.420106+0000","last_undegraded":"2024-03-26T02:01:49.420106+0000","last_fullsized":"2024-03-26T02:01:49.420106+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:58:32.416981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419745+0000","last_change":"2024-03-26T02:01:08.207724+0000","last_active":"2024-03-26T02:01:49.419745+0000","last_peered":"2024-03-26T02:01:49.419745+0000","last_clean":"2024-03-26T02:01:49.419745+0000","last_became_active":"2024-03-26T02:01:08.207198+0000","last_became_peered":"2024-03-26T02:01:08.207198+0000","last_unstale":"2024-03-26T02:01:49.419745+0000","last_undegraded":"2024-03-26T02:01:49.419745+0000","last_fullsized":"2024-03-26T02:01:49.419745+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:32:53.486843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'18","reported_seq":64,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419090+0000","last_change":"2024-03-26T02:01:10.226650+0000","last_active":"2024-03-26T02:01:49.419090+0000","last_peered":"2024-03-26T02:01:49.419090+0000","last_clean":"2024-03-26T02:01:49.419090+0000","last_became_active":"2024-03-26T02:01:10.224616+0000","last_became_peered":"2024-03-26T02:01:10.224616+0000","last_unstale":"2024-03-26T02:01:49.419090+0000","last_undegraded":"2024-03-26T02:01:49.419090+0000","last_fullsized":"2024-03-26T02:01:49.419090+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:05:28.121022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.427911+0000","last_change":"2024-03-26T02:01:14.250949+0000","last_active":"2024-03-26T02:01:49.427911+0000","last_peered":"2024-03-26T02:01:49.427911+0000","last_clean":"2024-03-26T02:01:49.427911+0000","last_became_active":"2024-03-26T02:01:14.250647+0000","last_became_peered":"2024-03-26T02:01:14.250647+0000","last_unstale":"2024-03-26T02:01:49.427911+0000","last_undegraded":"2024-03-26T02:01:49.427911+0000","last_fullsized":"2024-03-26T02:01:49.427911+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:55:59.190070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"67'8","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:02:18.262540+0000","last_change":"2024-03-26T02:01:12.244114+0000","last_active":"2024-03-26T02:02:18.262540+0000","last_peered":"2024-03-26T02:02:18.262540+0000","last_clean":"2024-03-26T02:02:18.262540+0000","last_became_active":"2024-03-26T02:01:12.243732+0000","last_became_peered":"2024-03-26T02:01:12.243732+0000","last_unstale":"2024-03-26T02:02:18.262540+0000","last_undegraded":"2024-03-26T02:02:18.262540+0000","last_fullsized":"2024-03-26T02:02:18.262540+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:41:52.651373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":34,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.738673+0000","last_change":"2024-03-26T02:01:08.212504+0000","last_active":"2024-03-26T02:01:49.738673+0000","last_peered":"2024-03-26T02:01:49.738673+0000","last_clean":"2024-03-26T02:01:49.738673+0000","last_became_active":"2024-03-26T02:01:08.212175+0000","last_became_peered":"2024-03-26T02:01:08.212175+0000","last_unstale":"2024-03-26T02:01:49.738673+0000","last_undegraded":"2024-03-26T02:01:49.738673+0000","last_fullsized":"2024-03-26T02:01:49.738673+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:20:33.512281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'16","reported_seq":56,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739660+0000","last_change":"2024-03-26T02:01:10.220706+0000","last_active":"2024-03-26T02:01:49.739660+0000","last_peered":"2024-03-26T02:01:49.739660+0000","last_clean":"2024-03-26T02:01:49.739660+0000","last_became_active":"2024-03-26T02:01:10.220421+0000","last_became_peered":"2024-03-26T02:01:10.220421+0000","last_unstale":"2024-03-26T02:01:49.739660+0000","last_undegraded":"2024-03-26T02:01:49.739660+0000","last_fullsized":"2024-03-26T02:01:49.739660+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:58:04.956823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.738066+0000","last_change":"2024-03-26T02:01:14.256824+0000","last_active":"2024-03-26T02:01:49.738066+0000","last_peered":"2024-03-26T02:01:49.738066+0000","last_clean":"2024-03-26T02:01:49.738066+0000","last_became_active":"2024-03-26T02:01:14.256483+0000","last_became_peered":"2024-03-26T02:01:14.256483+0000","last_unstale":"2024-03-26T02:01:49.738066+0000","last_undegraded":"2024-03-26T02:01:49.738066+0000","last_fullsized":"2024-03-26T02:01:49.738066+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:35:33.068064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739859+0000","last_change":"2024-03-26T02:01:12.247491+0000","last_active":"2024-03-26T02:01:49.739859+0000","last_peered":"2024-03-26T02:01:49.739859+0000","last_clean":"2024-03-26T02:01:49.739859+0000","last_became_active":"2024-03-26T02:01:12.247172+0000","last_became_peered":"2024-03-26T02:01:12.247172+0000","last_unstale":"2024-03-26T02:01:49.739859+0000","last_undegraded":"2024-03-26T02:01:49.739859+0000","last_fullsized":"2024-03-26T02:01:49.739859+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:24:12.549870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.418343+0000","last_change":"2024-03-26T02:01:08.209079+0000","last_active":"2024-03-26T02:01:49.418343+0000","last_peered":"2024-03-26T02:01:49.418343+0000","last_clean":"2024-03-26T02:01:49.418343+0000","last_became_active":"2024-03-26T02:01:08.207998+0000","last_became_peered":"2024-03-26T02:01:08.207998+0000","last_unstale":"2024-03-26T02:01:49.418343+0000","last_undegraded":"2024-03-26T02:01:49.418343+0000","last_fullsized":"2024-03-26T02:01:49.418343+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:16:59.366565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'17","reported_seq":59,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.595048+0000","last_change":"2024-03-26T02:01:10.221845+0000","last_active":"2024-03-26T02:01:49.595048+0000","last_peered":"2024-03-26T02:01:49.595048+0000","last_clean":"2024-03-26T02:01:49.595048+0000","last_became_active":"2024-03-26T02:01:10.221514+0000","last_became_peered":"2024-03-26T02:01:10.221514+0000","last_unstale":"2024-03-26T02:01:49.595048+0000","last_undegraded":"2024-03-26T02:01:49.595048+0000","last_fullsized":"2024-03-26T02:01:49.595048+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:20:23.001489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.427681+0000","last_change":"2024-03-26T02:01:14.256989+0000","last_active":"2024-03-26T02:01:49.427681+0000","last_peered":"2024-03-26T02:01:49.427681+0000","last_clean":"2024-03-26T02:01:49.427681+0000","last_became_active":"2024-03-26T02:01:14.256633+0000","last_became_peered":"2024-03-26T02:01:14.256633+0000","last_unstale":"2024-03-26T02:01:49.427681+0000","last_undegraded":"2024-03-26T02:01:49.427681+0000","last_fullsized":"2024-03-26T02:01:49.427681+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:55:24.654808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":24,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.594352+0000","last_change":"2024-03-26T02:01:12.242199+0000","last_active":"2024-03-26T02:01:49.594352+0000","last_peered":"2024-03-26T02:01:49.594352+0000","last_clean":"2024-03-26T02:01:49.594352+0000","last_became_active":"2024-03-26T02:01:12.241923+0000","last_became_peered":"2024-03-26T02:01:12.241923+0000","last_unstale":"2024-03-26T02:01:49.594352+0000","last_undegraded":"2024-03-26T02:01:49.594352+0000","last_fullsized":"2024-03-26T02:01:49.594352+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:55:48.612436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739198+0000","last_change":"2024-03-26T02:01:08.208640+0000","last_active":"2024-03-26T02:01:49.739198+0000","last_peered":"2024-03-26T02:01:49.739198+0000","last_clean":"2024-03-26T02:01:49.739198+0000","last_became_active":"2024-03-26T02:01:08.208365+0000","last_became_peered":"2024-03-26T02:01:08.208365+0000","last_unstale":"2024-03-26T02:01:49.739198+0000","last_undegraded":"2024-03-26T02:01:49.739198+0000","last_fullsized":"2024-03-26T02:01:49.739198+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:09:12.148252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":35,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.418425+0000","last_change":"2024-03-26T02:01:10.223832+0000","last_active":"2024-03-26T02:01:49.418425+0000","last_peered":"2024-03-26T02:01:49.418425+0000","last_clean":"2024-03-26T02:01:49.418425+0000","last_became_active":"2024-03-26T02:01:10.223496+0000","last_became_peered":"2024-03-26T02:01:10.223496+0000","last_unstale":"2024-03-26T02:01:49.418425+0000","last_undegraded":"2024-03-26T02:01:49.418425+0000","last_fullsized":"2024-03-26T02:01:49.418425+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:10:07.203212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"67'1","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419383+0000","last_change":"2024-03-26T02:01:14.949310+0000","last_active":"2024-03-26T02:01:49.419383+0000","last_peered":"2024-03-26T02:01:49.419383+0000","last_clean":"2024-03-26T02:01:49.419383+0000","last_became_active":"2024-03-26T02:01:14.948814+0000","last_became_peered":"2024-03-26T02:01:14.948814+0000","last_unstale":"2024-03-26T02:01:49.419383+0000","last_undegraded":"2024-03-26T02:01:49.419383+0000","last_fullsized":"2024-03-26T02:01:49.419383+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:43:59.999251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.427970+0000","last_change":"2024-03-26T02:01:12.245655+0000","last_active":"2024-03-26T02:01:49.427970+0000","last_peered":"2024-03-26T02:01:49.427970+0000","last_clean":"2024-03-26T02:01:49.427970+0000","last_became_active":"2024-03-26T02:01:12.245388+0000","last_became_peered":"2024-03-26T02:01:12.245388+0000","last_unstale":"2024-03-26T02:01:49.427970+0000","last_undegraded":"2024-03-26T02:01:49.427970+0000","last_fullsized":"2024-03-26T02:01:49.427970+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:14:14.607316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.534195+0000","last_change":"2024-03-26T02:01:08.211997+0000","last_active":"2024-03-26T02:01:49.534195+0000","last_peered":"2024-03-26T02:01:49.534195+0000","last_clean":"2024-03-26T02:01:49.534195+0000","last_became_active":"2024-03-26T02:01:08.211528+0000","last_became_peered":"2024-03-26T02:01:08.211528+0000","last_unstale":"2024-03-26T02:01:49.534195+0000","last_undegraded":"2024-03-26T02:01:49.534195+0000","last_fullsized":"2024-03-26T02:01:49.534195+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:55:24.339095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"67'17","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.418680+0000","last_change":"2024-03-26T02:01:10.224403+0000","last_active":"2024-03-26T02:01:49.418680+0000","last_peered":"2024-03-26T02:01:49.418680+0000","last_clean":"2024-03-26T02:01:49.418680+0000","last_became_active":"2024-03-26T02:01:10.222828+0000","last_became_peered":"2024-03-26T02:01:10.222828+0000","last_unstale":"2024-03-26T02:01:49.418680+0000","last_undegraded":"2024-03-26T02:01:49.418680+0000","last_fullsized":"2024-03-26T02:01:49.418680+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:52:50.179227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.737634+0000","last_change":"2024-03-26T02:01:14.251801+0000","last_active":"2024-03-26T02:01:49.737634+0000","last_peered":"2024-03-26T02:01:49.737634+0000","last_clean":"2024-03-26T02:01:49.737634+0000","last_became_active":"2024-03-26T02:01:14.251481+0000","last_became_peered":"2024-03-26T02:01:14.251481+0000","last_unstale":"2024-03-26T02:01:49.737634+0000","last_undegraded":"2024-03-26T02:01:49.737634+0000","last_fullsized":"2024-03-26T02:01:49.737634+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:31:01.718454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419601+0000","last_change":"2024-03-26T02:01:12.242618+0000","last_active":"2024-03-26T02:01:49.419601+0000","last_peered":"2024-03-26T02:01:49.419601+0000","last_clean":"2024-03-26T02:01:49.419601+0000","last_became_active":"2024-03-26T02:01:12.242307+0000","last_became_peered":"2024-03-26T02:01:12.242307+0000","last_unstale":"2024-03-26T02:01:49.419601+0000","last_undegraded":"2024-03-26T02:01:49.419601+0000","last_fullsized":"2024-03-26T02:01:49.419601+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:28:38.852684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.428150+0000","last_change":"2024-03-26T02:01:08.214034+0000","last_active":"2024-03-26T02:01:49.428150+0000","last_peered":"2024-03-26T02:01:49.428150+0000","last_clean":"2024-03-26T02:01:49.428150+0000","last_became_active":"2024-03-26T02:01:08.213753+0000","last_became_peered":"2024-03-26T02:01:08.213753+0000","last_unstale":"2024-03-26T02:01:49.428150+0000","last_undegraded":"2024-03-26T02:01:49.428150+0000","last_fullsized":"2024-03-26T02:01:49.428150+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:33:54.305735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.738540+0000","last_change":"2024-03-26T02:01:10.224719+0000","last_active":"2024-03-26T02:01:49.738540+0000","last_peered":"2024-03-26T02:01:49.738540+0000","last_clean":"2024-03-26T02:01:49.738540+0000","last_became_active":"2024-03-26T02:01:10.224459+0000","last_became_peered":"2024-03-26T02:01:10.224459+0000","last_unstale":"2024-03-26T02:01:49.738540+0000","last_undegraded":"2024-03-26T02:01:49.738540+0000","last_fullsized":"2024-03-26T02:01:49.738540+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:13:50.902683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.418403+0000","last_change":"2024-03-26T02:01:14.255613+0000","last_active":"2024-03-26T02:01:49.418403+0000","last_peered":"2024-03-26T02:01:49.418403+0000","last_clean":"2024-03-26T02:01:49.418403+0000","last_became_active":"2024-03-26T02:01:14.255250+0000","last_became_peered":"2024-03-26T02:01:14.255250+0000","last_unstale":"2024-03-26T02:01:49.418403+0000","last_undegraded":"2024-03-26T02:01:49.418403+0000","last_fullsized":"2024-03-26T02:01:49.418403+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:25:12.743536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.738409+0000","last_change":"2024-03-26T02:01:12.241570+0000","last_active":"2024-03-26T02:01:49.738409+0000","last_peered":"2024-03-26T02:01:49.738409+0000","last_clean":"2024-03-26T02:01:49.738409+0000","last_became_active":"2024-03-26T02:01:12.241250+0000","last_became_peered":"2024-03-26T02:01:12.241250+0000","last_unstale":"2024-03-26T02:01:49.738409+0000","last_undegraded":"2024-03-26T02:01:49.738409+0000","last_fullsized":"2024-03-26T02:01:49.738409+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:06:35.375774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.420294+0000","last_change":"2024-03-26T02:01:08.207936+0000","last_active":"2024-03-26T02:01:49.420294+0000","last_peered":"2024-03-26T02:01:49.420294+0000","last_clean":"2024-03-26T02:01:49.420294+0000","last_became_active":"2024-03-26T02:01:08.207656+0000","last_became_peered":"2024-03-26T02:01:08.207656+0000","last_unstale":"2024-03-26T02:01:49.420294+0000","last_undegraded":"2024-03-26T02:01:49.420294+0000","last_fullsized":"2024-03-26T02:01:49.420294+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:14:26.153078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.420231+0000","last_change":"2024-03-26T02:01:10.226699+0000","last_active":"2024-03-26T02:01:49.420231+0000","last_peered":"2024-03-26T02:01:49.420231+0000","last_clean":"2024-03-26T02:01:49.420231+0000","last_became_active":"2024-03-26T02:01:10.226402+0000","last_became_peered":"2024-03-26T02:01:10.226402+0000","last_unstale":"2024-03-26T02:01:49.420231+0000","last_undegraded":"2024-03-26T02:01:49.420231+0000","last_fullsized":"2024-03-26T02:01:49.420231+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:47:27.617674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739385+0000","last_change":"2024-03-26T02:01:14.253866+0000","last_active":"2024-03-26T02:01:49.739385+0000","last_peered":"2024-03-26T02:01:49.739385+0000","last_clean":"2024-03-26T02:01:49.739385+0000","last_became_active":"2024-03-26T02:01:14.253565+0000","last_became_peered":"2024-03-26T02:01:14.253565+0000","last_unstale":"2024-03-26T02:01:49.739385+0000","last_undegraded":"2024-03-26T02:01:49.739385+0000","last_fullsized":"2024-03-26T02:01:49.739385+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:30:07.931245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:02:18.262559+0000","last_change":"2024-03-26T02:01:12.244489+0000","last_active":"2024-03-26T02:02:18.262559+0000","last_peered":"2024-03-26T02:02:18.262559+0000","last_clean":"2024-03-26T02:02:18.262559+0000","last_became_active":"2024-03-26T02:01:12.244214+0000","last_became_peered":"2024-03-26T02:01:12.244214+0000","last_unstale":"2024-03-26T02:02:18.262559+0000","last_undegraded":"2024-03-26T02:02:18.262559+0000","last_fullsized":"2024-03-26T02:02:18.262559+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:48:08.608031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.427747+0000","last_change":"2024-03-26T02:01:08.215223+0000","last_active":"2024-03-26T02:01:49.427747+0000","last_peered":"2024-03-26T02:01:49.427747+0000","last_clean":"2024-03-26T02:01:49.427747+0000","last_became_active":"2024-03-26T02:01:08.214934+0000","last_became_peered":"2024-03-26T02:01:08.214934+0000","last_unstale":"2024-03-26T02:01:49.427747+0000","last_undegraded":"2024-03-26T02:01:49.427747+0000","last_fullsized":"2024-03-26T02:01:49.427747+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:38:54.711852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"67'22","reported_seq":90,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419032+0000","last_change":"2024-03-26T02:01:28.905784+0000","last_active":"2024-03-26T02:01:49.419032+0000","last_peered":"2024-03-26T02:01:49.419032+0000","last_clean":"2024-03-26T02:01:49.419032+0000","last_became_active":"2024-03-26T02:01:28.897223+0000","last_became_peered":"2024-03-26T02:01:28.897223+0000","last_unstale":"2024-03-26T02:01:49.419032+0000","last_undegraded":"2024-03-26T02:01:49.419032+0000","last_fullsized":"2024-03-26T02:01:49.419032+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:58:57.814858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.420997+0000","last_change":"2024-03-26T02:01:14.251114+0000","last_active":"2024-03-26T02:01:49.420997+0000","last_peered":"2024-03-26T02:01:49.420997+0000","last_clean":"2024-03-26T02:01:49.420997+0000","last_became_active":"2024-03-26T02:01:14.250786+0000","last_became_peered":"2024-03-26T02:01:14.250786+0000","last_unstale":"2024-03-26T02:01:49.420997+0000","last_undegraded":"2024-03-26T02:01:49.420997+0000","last_fullsized":"2024-03-26T02:01:49.420997+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:11:03.970677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:02:18.262437+0000","last_change":"2024-03-26T02:01:12.238242+0000","last_active":"2024-03-26T02:02:18.262437+0000","last_peered":"2024-03-26T02:02:18.262437+0000","last_clean":"2024-03-26T02:02:18.262437+0000","last_became_active":"2024-03-26T02:01:12.237645+0000","last_became_peered":"2024-03-26T02:01:12.237645+0000","last_unstale":"2024-03-26T02:02:18.262437+0000","last_undegraded":"2024-03-26T02:02:18.262437+0000","last_fullsized":"2024-03-26T02:02:18.262437+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:03:39.764784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.738434+0000","last_change":"2024-03-26T02:01:08.209746+0000","last_active":"2024-03-26T02:01:49.738434+0000","last_peered":"2024-03-26T02:01:49.738434+0000","last_clean":"2024-03-26T02:01:49.738434+0000","last_became_active":"2024-03-26T02:01:08.209459+0000","last_became_peered":"2024-03-26T02:01:08.209459+0000","last_unstale":"2024-03-26T02:01:49.738434+0000","last_undegraded":"2024-03-26T02:01:49.738434+0000","last_fullsized":"2024-03-26T02:01:49.738434+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:21:34.155937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"67'15","reported_seq":57,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.534270+0000","last_change":"2024-03-26T02:01:10.223496+0000","last_active":"2024-03-26T02:01:49.534270+0000","last_peered":"2024-03-26T02:01:49.534270+0000","last_clean":"2024-03-26T02:01:49.534270+0000","last_became_active":"2024-03-26T02:01:10.223197+0000","last_became_peered":"2024-03-26T02:01:10.223197+0000","last_unstale":"2024-03-26T02:01:49.534270+0000","last_undegraded":"2024-03-26T02:01:49.534270+0000","last_fullsized":"2024-03-26T02:01:49.534270+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:46:50.071363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.422478+0000","last_change":"2024-03-26T02:01:14.250352+0000","last_active":"2024-03-26T02:01:49.422478+0000","last_peered":"2024-03-26T02:01:49.422478+0000","last_clean":"2024-03-26T02:01:49.422478+0000","last_became_active":"2024-03-26T02:01:14.250046+0000","last_became_peered":"2024-03-26T02:01:14.250046+0000","last_unstale":"2024-03-26T02:01:49.422478+0000","last_undegraded":"2024-03-26T02:01:49.422478+0000","last_fullsized":"2024-03-26T02:01:49.422478+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:13:25.433809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.420171+0000","last_change":"2024-03-26T02:01:12.241307+0000","last_active":"2024-03-26T02:01:49.420171+0000","last_peered":"2024-03-26T02:01:49.420171+0000","last_clean":"2024-03-26T02:01:49.420171+0000","last_became_active":"2024-03-26T02:01:12.241004+0000","last_became_peered":"2024-03-26T02:01:12.241004+0000","last_unstale":"2024-03-26T02:01:49.420171+0000","last_undegraded":"2024-03-26T02:01:49.420171+0000","last_fullsized":"2024-03-26T02:01:49.420171+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:42:28.224971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.427791+0000","last_change":"2024-03-26T02:01:08.213532+0000","last_active":"2024-03-26T02:01:49.427791+0000","last_peered":"2024-03-26T02:01:49.427791+0000","last_clean":"2024-03-26T02:01:49.427791+0000","last_became_active":"2024-03-26T02:01:08.212567+0000","last_became_peered":"2024-03-26T02:01:08.212567+0000","last_unstale":"2024-03-26T02:01:49.427791+0000","last_undegraded":"2024-03-26T02:01:49.427791+0000","last_fullsized":"2024-03-26T02:01:49.427791+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:46:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'23","reported_seq":71,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739777+0000","last_change":"2024-03-26T02:01:10.220378+0000","last_active":"2024-03-26T02:01:49.739777+0000","last_peered":"2024-03-26T02:01:49.739777+0000","last_clean":"2024-03-26T02:01:49.739777+0000","last_became_active":"2024-03-26T02:01:10.220089+0000","last_became_peered":"2024-03-26T02:01:10.220089+0000","last_unstale":"2024-03-26T02:01:49.739777+0000","last_undegraded":"2024-03-26T02:01:49.739777+0000","last_fullsized":"2024-03-26T02:01:49.739777+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:10:13.561692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.428146+0000","last_change":"2024-03-26T02:01:14.949065+0000","last_active":"2024-03-26T02:01:49.428146+0000","last_peered":"2024-03-26T02:01:49.428146+0000","last_clean":"2024-03-26T02:01:49.428146+0000","last_became_active":"2024-03-26T02:01:14.948757+0000","last_became_peered":"2024-03-26T02:01:14.948757+0000","last_unstale":"2024-03-26T02:01:49.428146+0000","last_undegraded":"2024-03-26T02:01:49.428146+0000","last_fullsized":"2024-03-26T02:01:49.428146+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:52:26.089697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.534200+0000","last_change":"2024-03-26T02:01:12.243768+0000","last_active":"2024-03-26T02:01:49.534200+0000","last_peered":"2024-03-26T02:01:49.534200+0000","last_clean":"2024-03-26T02:01:49.534200+0000","last_became_active":"2024-03-26T02:01:12.243495+0000","last_became_peered":"2024-03-26T02:01:12.243495+0000","last_unstale":"2024-03-26T02:01:49.534200+0000","last_undegraded":"2024-03-26T02:01:49.534200+0000","last_fullsized":"2024-03-26T02:01:49.534200+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:31:48.783807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":32,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.594280+0000","last_change":"2024-03-26T02:01:08.214330+0000","last_active":"2024-03-26T02:01:49.594280+0000","last_peered":"2024-03-26T02:01:49.594280+0000","last_clean":"2024-03-26T02:01:49.594280+0000","last_became_active":"2024-03-26T02:01:08.214032+0000","last_became_peered":"2024-03-26T02:01:08.214032+0000","last_unstale":"2024-03-26T02:01:49.594280+0000","last_undegraded":"2024-03-26T02:01:49.594280+0000","last_fullsized":"2024-03-26T02:01:49.594280+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:49:32.455174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'12","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.420254+0000","last_change":"2024-03-26T02:01:10.226283+0000","last_active":"2024-03-26T02:01:49.420254+0000","last_peered":"2024-03-26T02:01:49.420254+0000","last_clean":"2024-03-26T02:01:49.420254+0000","last_became_active":"2024-03-26T02:01:10.224813+0000","last_became_peered":"2024-03-26T02:01:10.224813+0000","last_unstale":"2024-03-26T02:01:49.420254+0000","last_undegraded":"2024-03-26T02:01:49.420254+0000","last_fullsized":"2024-03-26T02:01:49.420254+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:38:22.316897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.533812+0000","last_change":"2024-03-26T02:01:14.949178+0000","last_active":"2024-03-26T02:01:49.533812+0000","last_peered":"2024-03-26T02:01:49.533812+0000","last_clean":"2024-03-26T02:01:49.533812+0000","last_became_active":"2024-03-26T02:01:14.948873+0000","last_became_peered":"2024-03-26T02:01:14.948873+0000","last_unstale":"2024-03-26T02:01:49.533812+0000","last_undegraded":"2024-03-26T02:01:49.533812+0000","last_fullsized":"2024-03-26T02:01:49.533812+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:31:50.292510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.428411+0000","last_change":"2024-03-26T02:01:08.215864+0000","last_active":"2024-03-26T02:01:49.428411+0000","last_peered":"2024-03-26T02:01:49.428411+0000","last_clean":"2024-03-26T02:01:49.428411+0000","last_became_active":"2024-03-26T02:01:08.215567+0000","last_became_peered":"2024-03-26T02:01:08.215567+0000","last_unstale":"2024-03-26T02:01:49.428411+0000","last_undegraded":"2024-03-26T02:01:49.428411+0000","last_fullsized":"2024-03-26T02:01:49.428411+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:42:59.899652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"67'29","reported_seq":80,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.533470+0000","last_change":"2024-03-26T02:01:10.220906+0000","last_active":"2024-03-26T02:01:49.533470+0000","last_peered":"2024-03-26T02:01:49.533470+0000","last_clean":"2024-03-26T02:01:49.533470+0000","last_became_active":"2024-03-26T02:01:10.220534+0000","last_became_peered":"2024-03-26T02:01:10.220534+0000","last_unstale":"2024-03-26T02:01:49.533470+0000","last_undegraded":"2024-03-26T02:01:49.533470+0000","last_fullsized":"2024-03-26T02:01:49.533470+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:05:56.519589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"67'8","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:02:18.262600+0000","last_change":"2024-03-26T02:01:12.246468+0000","last_active":"2024-03-26T02:02:18.262600+0000","last_peered":"2024-03-26T02:02:18.262600+0000","last_clean":"2024-03-26T02:02:18.262600+0000","last_became_active":"2024-03-26T02:01:12.246168+0000","last_became_peered":"2024-03-26T02:01:12.246168+0000","last_unstale":"2024-03-26T02:02:18.262600+0000","last_undegraded":"2024-03-26T02:02:18.262600+0000","last_fullsized":"2024-03-26T02:02:18.262600+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:17:48.341193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419388+0000","last_change":"2024-03-26T02:01:14.258131+0000","last_active":"2024-03-26T02:01:49.419388+0000","last_peered":"2024-03-26T02:01:49.419388+0000","last_clean":"2024-03-26T02:01:49.419388+0000","last_became_active":"2024-03-26T02:01:14.257378+0000","last_became_peered":"2024-03-26T02:01:14.257378+0000","last_unstale":"2024-03-26T02:01:49.419388+0000","last_undegraded":"2024-03-26T02:01:49.419388+0000","last_fullsized":"2024-03-26T02:01:49.419388+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:10:51.748424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.428359+0000","last_change":"2024-03-26T02:01:08.217120+0000","last_active":"2024-03-26T02:01:49.428359+0000","last_peered":"2024-03-26T02:01:49.428359+0000","last_clean":"2024-03-26T02:01:49.428359+0000","last_became_active":"2024-03-26T02:01:08.216554+0000","last_became_peered":"2024-03-26T02:01:08.216554+0000","last_unstale":"2024-03-26T02:01:49.428359+0000","last_undegraded":"2024-03-26T02:01:49.428359+0000","last_fullsized":"2024-03-26T02:01:49.428359+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:54:25.381293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"67'24","reported_seq":98,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.420393+0000","last_change":"2024-03-26T02:01:28.912740+0000","last_active":"2024-03-26T02:01:49.420393+0000","last_peered":"2024-03-26T02:01:49.420393+0000","last_clean":"2024-03-26T02:01:49.420393+0000","last_became_active":"2024-03-26T02:01:28.898207+0000","last_became_peered":"2024-03-26T02:01:28.898207+0000","last_unstale":"2024-03-26T02:01:49.420393+0000","last_undegraded":"2024-03-26T02:01:49.420393+0000","last_fullsized":"2024-03-26T02:01:49.420393+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:27:06.140652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.421536+0000","last_change":"2024-03-26T02:01:12.239996+0000","last_active":"2024-03-26T02:01:49.421536+0000","last_peered":"2024-03-26T02:01:49.421536+0000","last_clean":"2024-03-26T02:01:49.421536+0000","last_became_active":"2024-03-26T02:01:12.239686+0000","last_became_peered":"2024-03-26T02:01:12.239686+0000","last_unstale":"2024-03-26T02:01:49.421536+0000","last_undegraded":"2024-03-26T02:01:49.421536+0000","last_fullsized":"2024-03-26T02:01:49.421536+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:50:07.924894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.427565+0000","last_change":"2024-03-26T02:01:14.259206+0000","last_active":"2024-03-26T02:01:49.427565+0000","last_peered":"2024-03-26T02:01:49.427565+0000","last_clean":"2024-03-26T02:01:49.427565+0000","last_became_active":"2024-03-26T02:01:14.258910+0000","last_became_peered":"2024-03-26T02:01:14.258910+0000","last_unstale":"2024-03-26T02:01:49.427565+0000","last_undegraded":"2024-03-26T02:01:49.427565+0000","last_fullsized":"2024-03-26T02:01:49.427565+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:43:01.661996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.421911+0000","last_change":"2024-03-26T02:01:12.242143+0000","last_active":"2024-03-26T02:01:49.421911+0000","last_peered":"2024-03-26T02:01:49.421911+0000","last_clean":"2024-03-26T02:01:49.421911+0000","last_became_active":"2024-03-26T02:01:12.241815+0000","last_became_peered":"2024-03-26T02:01:12.241815+0000","last_unstale":"2024-03-26T02:01:49.421911+0000","last_undegraded":"2024-03-26T02:01:49.421911+0000","last_fullsized":"2024-03-26T02:01:49.421911+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:13:32.222462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"60'1","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.428356+0000","last_change":"2024-03-26T02:01:08.213420+0000","last_active":"2024-03-26T02:01:49.428356+0000","last_peered":"2024-03-26T02:01:49.428356+0000","last_clean":"2024-03-26T02:01:49.428356+0000","last_became_active":"2024-03-26T02:01:08.213157+0000","last_became_peered":"2024-03-26T02:01:08.213157+0000","last_unstale":"2024-03-26T02:01:49.428356+0000","last_undegraded":"2024-03-26T02:01:49.428356+0000","last_fullsized":"2024-03-26T02:01:49.428356+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:02:06.787189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"67'25","reported_seq":71,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.594781+0000","last_change":"2024-03-26T02:01:10.222552+0000","last_active":"2024-03-26T02:01:49.594781+0000","last_peered":"2024-03-26T02:01:49.594781+0000","last_clean":"2024-03-26T02:01:49.594781+0000","last_became_active":"2024-03-26T02:01:10.221462+0000","last_became_peered":"2024-03-26T02:01:10.221462+0000","last_unstale":"2024-03-26T02:01:49.594781+0000","last_undegraded":"2024-03-26T02:01:49.594781+0000","last_fullsized":"2024-03-26T02:01:49.594781+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:51:41.373836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.737745+0000","last_change":"2024-03-26T02:01:14.248293+0000","last_active":"2024-03-26T02:01:49.737745+0000","last_peered":"2024-03-26T02:01:49.737745+0000","last_clean":"2024-03-26T02:01:49.737745+0000","last_became_active":"2024-03-26T02:01:14.247954+0000","last_became_peered":"2024-03-26T02:01:14.247954+0000","last_unstale":"2024-03-26T02:01:49.737745+0000","last_undegraded":"2024-03-26T02:01:49.737745+0000","last_fullsized":"2024-03-26T02:01:49.737745+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:11:35.383448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.421962+0000","last_change":"2024-03-26T02:01:12.235568+0000","last_active":"2024-03-26T02:01:49.421962+0000","last_peered":"2024-03-26T02:01:49.421962+0000","last_clean":"2024-03-26T02:01:49.421962+0000","last_became_active":"2024-03-26T02:01:12.235220+0000","last_became_peered":"2024-03-26T02:01:12.235220+0000","last_unstale":"2024-03-26T02:01:49.421962+0000","last_undegraded":"2024-03-26T02:01:49.421962+0000","last_fullsized":"2024-03-26T02:01:49.421962+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:43:44.371930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.533892+0000","last_change":"2024-03-26T02:01:08.207634+0000","last_active":"2024-03-26T02:01:49.533892+0000","last_peered":"2024-03-26T02:01:49.533892+0000","last_clean":"2024-03-26T02:01:49.533892+0000","last_became_active":"2024-03-26T02:01:08.207339+0000","last_became_peered":"2024-03-26T02:01:08.207339+0000","last_unstale":"2024-03-26T02:01:49.533892+0000","last_undegraded":"2024-03-26T02:01:49.533892+0000","last_fullsized":"2024-03-26T02:01:49.533892+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:18:59.825271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"67'16","reported_seq":58,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739868+0000","last_change":"2024-03-26T02:01:10.221814+0000","last_active":"2024-03-26T02:01:49.739868+0000","last_peered":"2024-03-26T02:01:49.739868+0000","last_clean":"2024-03-26T02:01:49.739868+0000","last_became_active":"2024-03-26T02:01:10.221512+0000","last_became_peered":"2024-03-26T02:01:10.221512+0000","last_unstale":"2024-03-26T02:01:49.739868+0000","last_undegraded":"2024-03-26T02:01:49.739868+0000","last_fullsized":"2024-03-26T02:01:49.739868+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:26:41.564898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419423+0000","last_change":"2024-03-26T02:01:14.254416+0000","last_active":"2024-03-26T02:01:49.419423+0000","last_peered":"2024-03-26T02:01:49.419423+0000","last_clean":"2024-03-26T02:01:49.419423+0000","last_became_active":"2024-03-26T02:01:14.254096+0000","last_became_peered":"2024-03-26T02:01:14.254096+0000","last_unstale":"2024-03-26T02:01:49.419423+0000","last_undegraded":"2024-03-26T02:01:49.419423+0000","last_fullsized":"2024-03-26T02:01:49.419423+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:42:00.911783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":42,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.533603+0000","last_change":"2024-03-26T02:01:12.234741+0000","last_active":"2024-03-26T02:01:49.533603+0000","last_peered":"2024-03-26T02:01:49.533603+0000","last_clean":"2024-03-26T02:01:49.533603+0000","last_became_active":"2024-03-26T02:01:08.212363+0000","last_became_peered":"2024-03-26T02:01:08.212363+0000","last_unstale":"2024-03-26T02:01:49.533603+0000","last_undegraded":"2024-03-26T02:01:49.533603+0000","last_fullsized":"2024-03-26T02:01:49.533603+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:56:33.744663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000429784,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419670+0000","last_change":"2024-03-26T02:01:08.210938+0000","last_active":"2024-03-26T02:01:49.419670+0000","last_peered":"2024-03-26T02:01:49.419670+0000","last_clean":"2024-03-26T02:01:49.419670+0000","last_became_active":"2024-03-26T02:01:08.210662+0000","last_became_peered":"2024-03-26T02:01:08.210662+0000","last_unstale":"2024-03-26T02:01:49.419670+0000","last_undegraded":"2024-03-26T02:01:49.419670+0000","last_fullsized":"2024-03-26T02:01:49.419670+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:11:37.586828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"67'34","reported_seq":105,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:02:18.262474+0000","last_change":"2024-03-26T02:01:10.228402+0000","last_active":"2024-03-26T02:02:18.262474+0000","last_peered":"2024-03-26T02:02:18.262474+0000","last_clean":"2024-03-26T02:02:18.262474+0000","last_became_active":"2024-03-26T02:01:10.227164+0000","last_became_peered":"2024-03-26T02:01:10.227164+0000","last_unstale":"2024-03-26T02:02:18.262474+0000","last_undegraded":"2024-03-26T02:02:18.262474+0000","last_fullsized":"2024-03-26T02:02:18.262474+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:05:11.832807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.738797+0000","last_change":"2024-03-26T02:01:12.245857+0000","last_active":"2024-03-26T02:01:49.738797+0000","last_peered":"2024-03-26T02:01:49.738797+0000","last_clean":"2024-03-26T02:01:49.738797+0000","last_became_active":"2024-03-26T02:01:12.245552+0000","last_became_peered":"2024-03-26T02:01:12.245552+0000","last_unstale":"2024-03-26T02:01:49.738797+0000","last_undegraded":"2024-03-26T02:01:49.738797+0000","last_fullsized":"2024-03-26T02:01:49.738797+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:53:04.495194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.428456+0000","last_change":"2024-03-26T02:01:14.947311+0000","last_active":"2024-03-26T02:01:49.428456+0000","last_peered":"2024-03-26T02:01:49.428456+0000","last_clean":"2024-03-26T02:01:49.428456+0000","last_became_active":"2024-03-26T02:01:14.947010+0000","last_became_peered":"2024-03-26T02:01:14.947010+0000","last_unstale":"2024-03-26T02:01:49.428456+0000","last_undegraded":"2024-03-26T02:01:49.428456+0000","last_fullsized":"2024-03-26T02:01:49.428456+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:12:57.418404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739224+0000","last_change":"2024-03-26T02:01:08.213992+0000","last_active":"2024-03-26T02:01:49.739224+0000","last_peered":"2024-03-26T02:01:49.739224+0000","last_clean":"2024-03-26T02:01:49.739224+0000","last_became_active":"2024-03-26T02:01:08.213716+0000","last_became_peered":"2024-03-26T02:01:08.213716+0000","last_unstale":"2024-03-26T02:01:49.739224+0000","last_undegraded":"2024-03-26T02:01:49.739224+0000","last_fullsized":"2024-03-26T02:01:49.739224+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:37:06.398832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.421692+0000","last_change":"2024-03-26T02:01:12.230250+0000","last_active":"2024-03-26T02:01:49.421692+0000","last_peered":"2024-03-26T02:01:49.421692+0000","last_clean":"2024-03-26T02:01:49.421692+0000","last_became_active":"2024-03-26T02:01:08.205688+0000","last_became_peered":"2024-03-26T02:01:08.205688+0000","last_unstale":"2024-03-26T02:01:49.421692+0000","last_undegraded":"2024-03-26T02:01:49.421692+0000","last_fullsized":"2024-03-26T02:01:49.421692+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:58:48.031942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498044,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"67'19","reported_seq":68,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.738996+0000","last_change":"2024-03-26T02:01:10.224787+0000","last_active":"2024-03-26T02:01:49.738996+0000","last_peered":"2024-03-26T02:01:49.738996+0000","last_clean":"2024-03-26T02:01:49.738996+0000","last_became_active":"2024-03-26T02:01:10.223876+0000","last_became_peered":"2024-03-26T02:01:10.223876+0000","last_unstale":"2024-03-26T02:01:49.738996+0000","last_undegraded":"2024-03-26T02:01:49.738996+0000","last_fullsized":"2024-03-26T02:01:49.738996+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:23:59.697926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.422065+0000","last_change":"2024-03-26T02:01:12.240414+0000","last_active":"2024-03-26T02:01:49.422065+0000","last_peered":"2024-03-26T02:01:49.422065+0000","last_clean":"2024-03-26T02:01:49.422065+0000","last_became_active":"2024-03-26T02:01:12.240118+0000","last_became_peered":"2024-03-26T02:01:12.240118+0000","last_unstale":"2024-03-26T02:01:49.422065+0000","last_undegraded":"2024-03-26T02:01:49.422065+0000","last_fullsized":"2024-03-26T02:01:49.422065+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:54:59.325332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.534217+0000","last_change":"2024-03-26T02:01:14.253928+0000","last_active":"2024-03-26T02:01:49.534217+0000","last_peered":"2024-03-26T02:01:49.534217+0000","last_clean":"2024-03-26T02:01:49.534217+0000","last_became_active":"2024-03-26T02:01:14.253594+0000","last_became_peered":"2024-03-26T02:01:14.253594+0000","last_unstale":"2024-03-26T02:01:49.534217+0000","last_undegraded":"2024-03-26T02:01:49.534217+0000","last_fullsized":"2024-03-26T02:01:49.534217+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:54:18.338047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"71'128","reported_seq":276,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:51.518377+0000","last_change":"2024-03-26T02:00:57.791957+0000","last_active":"2024-03-26T02:01:51.518377+0000","last_peered":"2024-03-26T02:01:51.518377+0000","last_clean":"2024-03-26T02:01:51.518377+0000","last_became_active":"2024-03-26T02:00:57.791412+0000","last_became_peered":"2024-03-26T02:00:57.791412+0000","last_unstale":"2024-03-26T02:01:51.518377+0000","last_undegraded":"2024-03-26T02:01:51.518377+0000","last_fullsized":"2024-03-26T02:01:51.518377+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:47.619613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:47.619613+0000","last_clean_scrub_stamp":"2024-03-26T01:58:47.619613+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:17:46.502230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419222+0000","last_change":"2024-03-26T02:01:08.212292+0000","last_active":"2024-03-26T02:01:49.419222+0000","last_peered":"2024-03-26T02:01:49.419222+0000","last_clean":"2024-03-26T02:01:49.419222+0000","last_became_active":"2024-03-26T02:01:08.211839+0000","last_became_peered":"2024-03-26T02:01:08.211839+0000","last_unstale":"2024-03-26T02:01:49.419222+0000","last_undegraded":"2024-03-26T02:01:49.419222+0000","last_fullsized":"2024-03-26T02:01:49.419222+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:10:02.050298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"67'16","reported_seq":66,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:02:18.262572+0000","last_change":"2024-03-26T02:01:10.223525+0000","last_active":"2024-03-26T02:02:18.262572+0000","last_peered":"2024-03-26T02:02:18.262572+0000","last_clean":"2024-03-26T02:02:18.262572+0000","last_became_active":"2024-03-26T02:01:10.223221+0000","last_became_peered":"2024-03-26T02:01:10.223221+0000","last_unstale":"2024-03-26T02:02:18.262572+0000","last_undegraded":"2024-03-26T02:02:18.262572+0000","last_fullsized":"2024-03-26T02:02:18.262572+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:41:59.197773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.428069+0000","last_change":"2024-03-26T02:01:12.241520+0000","last_active":"2024-03-26T02:01:49.428069+0000","last_peered":"2024-03-26T02:01:49.428069+0000","last_clean":"2024-03-26T02:01:49.428069+0000","last_became_active":"2024-03-26T02:01:12.241240+0000","last_became_peered":"2024-03-26T02:01:12.241240+0000","last_unstale":"2024-03-26T02:01:49.428069+0000","last_undegraded":"2024-03-26T02:01:49.428069+0000","last_fullsized":"2024-03-26T02:01:49.428069+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:22:44.698597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739190+0000","last_change":"2024-03-26T02:01:14.259371+0000","last_active":"2024-03-26T02:01:49.739190+0000","last_peered":"2024-03-26T02:01:49.739190+0000","last_clean":"2024-03-26T02:01:49.739190+0000","last_became_active":"2024-03-26T02:01:14.259028+0000","last_became_peered":"2024-03-26T02:01:14.259028+0000","last_unstale":"2024-03-26T02:01:49.739190+0000","last_undegraded":"2024-03-26T02:01:49.739190+0000","last_fullsized":"2024-03-26T02:01:49.739190+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:24:26.238493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"60'1","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739030+0000","last_change":"2024-03-26T02:01:08.211304+0000","last_active":"2024-03-26T02:01:49.739030+0000","last_peered":"2024-03-26T02:01:49.739030+0000","last_clean":"2024-03-26T02:01:49.739030+0000","last_became_active":"2024-03-26T02:01:08.210886+0000","last_became_peered":"2024-03-26T02:01:08.210886+0000","last_unstale":"2024-03-26T02:01:49.739030+0000","last_undegraded":"2024-03-26T02:01:49.739030+0000","last_fullsized":"2024-03-26T02:01:49.739030+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:11:45.385952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":114,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:02:24.346142+0000","last_change":"2024-03-26T02:01:12.234879+0000","last_active":"2024-03-26T02:02:24.346142+0000","last_peered":"2024-03-26T02:02:24.346142+0000","last_clean":"2024-03-26T02:02:24.346142+0000","last_became_active":"2024-03-26T02:01:08.215480+0000","last_became_peered":"2024-03-26T02:01:08.215480+0000","last_unstale":"2024-03-26T02:02:24.346142+0000","last_undegraded":"2024-03-26T02:02:24.346142+0000","last_fullsized":"2024-03-26T02:02:24.346142+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:29:30.600562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081101999999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":71,"num_read_kb":66,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":47,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739475+0000","last_change":"2024-03-26T02:01:10.222556+0000","last_active":"2024-03-26T02:01:49.739475+0000","last_peered":"2024-03-26T02:01:49.739475+0000","last_clean":"2024-03-26T02:01:49.739475+0000","last_became_active":"2024-03-26T02:01:10.222239+0000","last_became_peered":"2024-03-26T02:01:10.222239+0000","last_unstale":"2024-03-26T02:01:49.739475+0000","last_undegraded":"2024-03-26T02:01:49.739475+0000","last_fullsized":"2024-03-26T02:01:49.739475+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:39:23.747045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.534451+0000","last_change":"2024-03-26T02:01:12.241505+0000","last_active":"2024-03-26T02:01:49.534451+0000","last_peered":"2024-03-26T02:01:49.534451+0000","last_clean":"2024-03-26T02:01:49.534451+0000","last_became_active":"2024-03-26T02:01:12.241236+0000","last_became_peered":"2024-03-26T02:01:12.241236+0000","last_unstale":"2024-03-26T02:01:49.534451+0000","last_undegraded":"2024-03-26T02:01:49.534451+0000","last_fullsized":"2024-03-26T02:01:49.534451+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:05:19.694248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.737938+0000","last_change":"2024-03-26T02:01:14.947343+0000","last_active":"2024-03-26T02:01:49.737938+0000","last_peered":"2024-03-26T02:01:49.737938+0000","last_clean":"2024-03-26T02:01:49.737938+0000","last_became_active":"2024-03-26T02:01:14.946957+0000","last_became_peered":"2024-03-26T02:01:14.946957+0000","last_unstale":"2024-03-26T02:01:49.737938+0000","last_undegraded":"2024-03-26T02:01:49.737938+0000","last_fullsized":"2024-03-26T02:01:49.737938+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:06:19.792424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739435+0000","last_change":"2024-03-26T02:01:08.211850+0000","last_active":"2024-03-26T02:01:49.739435+0000","last_peered":"2024-03-26T02:01:49.739435+0000","last_clean":"2024-03-26T02:01:49.739435+0000","last_became_active":"2024-03-26T02:01:08.211515+0000","last_became_peered":"2024-03-26T02:01:08.211515+0000","last_unstale":"2024-03-26T02:01:49.739435+0000","last_undegraded":"2024-03-26T02:01:49.739435+0000","last_fullsized":"2024-03-26T02:01:49.739435+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:00:03.829850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"67'25","reported_seq":77,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739659+0000","last_change":"2024-03-26T02:01:10.217804+0000","last_active":"2024-03-26T02:01:49.739659+0000","last_peered":"2024-03-26T02:01:49.739659+0000","last_clean":"2024-03-26T02:01:49.739659+0000","last_became_active":"2024-03-26T02:01:10.217469+0000","last_became_peered":"2024-03-26T02:01:10.217469+0000","last_unstale":"2024-03-26T02:01:49.739659+0000","last_undegraded":"2024-03-26T02:01:49.739659+0000","last_fullsized":"2024-03-26T02:01:49.739659+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:10:20.623390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":24,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.594958+0000","last_change":"2024-03-26T02:01:12.241024+0000","last_active":"2024-03-26T02:01:49.594958+0000","last_peered":"2024-03-26T02:01:49.594958+0000","last_clean":"2024-03-26T02:01:49.594958+0000","last_became_active":"2024-03-26T02:01:12.240748+0000","last_became_peered":"2024-03-26T02:01:12.240748+0000","last_unstale":"2024-03-26T02:01:49.594958+0000","last_undegraded":"2024-03-26T02:01:49.594958+0000","last_fullsized":"2024-03-26T02:01:49.594958+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:47:52.744694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.737634+0000","last_change":"2024-03-26T02:01:14.259290+0000","last_active":"2024-03-26T02:01:49.737634+0000","last_peered":"2024-03-26T02:01:49.737634+0000","last_clean":"2024-03-26T02:01:49.737634+0000","last_became_active":"2024-03-26T02:01:14.258916+0000","last_became_peered":"2024-03-26T02:01:14.258916+0000","last_unstale":"2024-03-26T02:01:49.737634+0000","last_undegraded":"2024-03-26T02:01:49.737634+0000","last_fullsized":"2024-03-26T02:01:49.737634+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:19:29.730587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.534124+0000","last_change":"2024-03-26T02:01:08.213557+0000","last_active":"2024-03-26T02:01:49.534124+0000","last_peered":"2024-03-26T02:01:49.534124+0000","last_clean":"2024-03-26T02:01:49.534124+0000","last_became_active":"2024-03-26T02:01:08.213200+0000","last_became_peered":"2024-03-26T02:01:08.213200+0000","last_unstale":"2024-03-26T02:01:49.534124+0000","last_undegraded":"2024-03-26T02:01:49.534124+0000","last_fullsized":"2024-03-26T02:01:49.534124+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:54:26.117881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"67'16","reported_seq":56,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.738505+0000","last_change":"2024-03-26T02:01:10.227638+0000","last_active":"2024-03-26T02:01:49.738505+0000","last_peered":"2024-03-26T02:01:49.738505+0000","last_clean":"2024-03-26T02:01:49.738505+0000","last_became_active":"2024-03-26T02:01:10.227235+0000","last_became_peered":"2024-03-26T02:01:10.227235+0000","last_unstale":"2024-03-26T02:01:49.738505+0000","last_undegraded":"2024-03-26T02:01:49.738505+0000","last_fullsized":"2024-03-26T02:01:49.738505+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:01:21.288308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"67'8","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:02:18.262419+0000","last_change":"2024-03-26T02:01:12.247746+0000","last_active":"2024-03-26T02:02:18.262419+0000","last_peered":"2024-03-26T02:02:18.262419+0000","last_clean":"2024-03-26T02:02:18.262419+0000","last_became_active":"2024-03-26T02:01:12.247388+0000","last_became_peered":"2024-03-26T02:01:12.247388+0000","last_unstale":"2024-03-26T02:02:18.262419+0000","last_undegraded":"2024-03-26T02:02:18.262419+0000","last_fullsized":"2024-03-26T02:02:18.262419+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:20:09.567057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.428086+0000","last_change":"2024-03-26T02:01:14.950129+0000","last_active":"2024-03-26T02:01:49.428086+0000","last_peered":"2024-03-26T02:01:49.428086+0000","last_clean":"2024-03-26T02:01:49.428086+0000","last_became_active":"2024-03-26T02:01:14.949518+0000","last_became_peered":"2024-03-26T02:01:14.949518+0000","last_unstale":"2024-03-26T02:01:49.428086+0000","last_undegraded":"2024-03-26T02:01:49.428086+0000","last_fullsized":"2024-03-26T02:01:49.428086+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:32:44.073092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":34,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739660+0000","last_change":"2024-03-26T02:01:08.209583+0000","last_active":"2024-03-26T02:01:49.739660+0000","last_peered":"2024-03-26T02:01:49.739660+0000","last_clean":"2024-03-26T02:01:49.739660+0000","last_became_active":"2024-03-26T02:01:08.209301+0000","last_became_peered":"2024-03-26T02:01:08.209301+0000","last_unstale":"2024-03-26T02:01:49.739660+0000","last_undegraded":"2024-03-26T02:01:49.739660+0000","last_fullsized":"2024-03-26T02:01:49.739660+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:16:57.355869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"67'20","reported_seq":62,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419741+0000","last_change":"2024-03-26T02:01:10.221627+0000","last_active":"2024-03-26T02:01:49.419741+0000","last_peered":"2024-03-26T02:01:49.419741+0000","last_clean":"2024-03-26T02:01:49.419741+0000","last_became_active":"2024-03-26T02:01:10.221328+0000","last_became_peered":"2024-03-26T02:01:10.221328+0000","last_unstale":"2024-03-26T02:01:49.419741+0000","last_undegraded":"2024-03-26T02:01:49.419741+0000","last_fullsized":"2024-03-26T02:01:49.419741+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:13:13.086773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:02:18.262420+0000","last_change":"2024-03-26T02:01:12.240289+0000","last_active":"2024-03-26T02:02:18.262420+0000","last_peered":"2024-03-26T02:02:18.262420+0000","last_clean":"2024-03-26T02:02:18.262420+0000","last_became_active":"2024-03-26T02:01:12.239975+0000","last_became_peered":"2024-03-26T02:01:12.239975+0000","last_unstale":"2024-03-26T02:02:18.262420+0000","last_undegraded":"2024-03-26T02:02:18.262420+0000","last_fullsized":"2024-03-26T02:02:18.262420+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:15:45.672112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419594+0000","last_change":"2024-03-26T02:01:14.253107+0000","last_active":"2024-03-26T02:01:49.419594+0000","last_peered":"2024-03-26T02:01:49.419594+0000","last_clean":"2024-03-26T02:01:49.419594+0000","last_became_active":"2024-03-26T02:01:14.252756+0000","last_became_peered":"2024-03-26T02:01:14.252756+0000","last_unstale":"2024-03-26T02:01:49.419594+0000","last_undegraded":"2024-03-26T02:01:49.419594+0000","last_fullsized":"2024-03-26T02:01:49.419594+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:46:36.520189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419954+0000","last_change":"2024-03-26T02:01:08.213368+0000","last_active":"2024-03-26T02:01:49.419954+0000","last_peered":"2024-03-26T02:01:49.419954+0000","last_clean":"2024-03-26T02:01:49.419954+0000","last_became_active":"2024-03-26T02:01:08.213072+0000","last_became_peered":"2024-03-26T02:01:08.213072+0000","last_unstale":"2024-03-26T02:01:49.419954+0000","last_undegraded":"2024-03-26T02:01:49.419954+0000","last_fullsized":"2024-03-26T02:01:49.419954+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:37:00.672363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"67'30","reported_seq":85,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.418547+0000","last_change":"2024-03-26T02:01:10.222574+0000","last_active":"2024-03-26T02:01:49.418547+0000","last_peered":"2024-03-26T02:01:49.418547+0000","last_clean":"2024-03-26T02:01:49.418547+0000","last_became_active":"2024-03-26T02:01:10.220256+0000","last_became_peered":"2024-03-26T02:01:10.220256+0000","last_unstale":"2024-03-26T02:01:49.418547+0000","last_undegraded":"2024-03-26T02:01:49.418547+0000","last_fullsized":"2024-03-26T02:01:49.418547+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:03:18.794657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419437+0000","last_change":"2024-03-26T02:01:12.244042+0000","last_active":"2024-03-26T02:01:49.419437+0000","last_peered":"2024-03-26T02:01:49.419437+0000","last_clean":"2024-03-26T02:01:49.419437+0000","last_became_active":"2024-03-26T02:01:12.242628+0000","last_became_peered":"2024-03-26T02:01:12.242628+0000","last_unstale":"2024-03-26T02:01:49.419437+0000","last_undegraded":"2024-03-26T02:01:49.419437+0000","last_fullsized":"2024-03-26T02:01:49.419437+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:21:22.900048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.533913+0000","last_change":"2024-03-26T02:01:14.252827+0000","last_active":"2024-03-26T02:01:49.533913+0000","last_peered":"2024-03-26T02:01:49.533913+0000","last_clean":"2024-03-26T02:01:49.533913+0000","last_became_active":"2024-03-26T02:01:14.252528+0000","last_became_peered":"2024-03-26T02:01:14.252528+0000","last_unstale":"2024-03-26T02:01:49.533913+0000","last_undegraded":"2024-03-26T02:01:49.533913+0000","last_fullsized":"2024-03-26T02:01:49.533913+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:49:38.186864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:02:18.262556+0000","last_change":"2024-03-26T02:01:12.243196+0000","last_active":"2024-03-26T02:02:18.262556+0000","last_peered":"2024-03-26T02:02:18.262556+0000","last_clean":"2024-03-26T02:02:18.262556+0000","last_became_active":"2024-03-26T02:01:12.242907+0000","last_became_peered":"2024-03-26T02:01:12.242907+0000","last_unstale":"2024-03-26T02:02:18.262556+0000","last_undegraded":"2024-03-26T02:02:18.262556+0000","last_fullsized":"2024-03-26T02:02:18.262556+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:48:21.562936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.418523+0000","last_change":"2024-03-26T02:01:08.209782+0000","last_active":"2024-03-26T02:01:49.418523+0000","last_peered":"2024-03-26T02:01:49.418523+0000","last_clean":"2024-03-26T02:01:49.418523+0000","last_became_active":"2024-03-26T02:01:08.209487+0000","last_became_peered":"2024-03-26T02:01:08.209487+0000","last_unstale":"2024-03-26T02:01:49.418523+0000","last_undegraded":"2024-03-26T02:01:49.418523+0000","last_fullsized":"2024-03-26T02:01:49.418523+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:57:12.003049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739114+0000","last_change":"2024-03-26T02:01:10.227401+0000","last_active":"2024-03-26T02:01:49.739114+0000","last_peered":"2024-03-26T02:01:49.739114+0000","last_clean":"2024-03-26T02:01:49.739114+0000","last_became_active":"2024-03-26T02:01:10.227113+0000","last_became_peered":"2024-03-26T02:01:10.227113+0000","last_unstale":"2024-03-26T02:01:49.739114+0000","last_undegraded":"2024-03-26T02:01:49.739114+0000","last_fullsized":"2024-03-26T02:01:49.739114+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:57:37.814294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.418798+0000","last_change":"2024-03-26T02:01:14.947358+0000","last_active":"2024-03-26T02:01:49.418798+0000","last_peered":"2024-03-26T02:01:49.418798+0000","last_clean":"2024-03-26T02:01:49.418798+0000","last_became_active":"2024-03-26T02:01:14.946985+0000","last_became_peered":"2024-03-26T02:01:14.946985+0000","last_unstale":"2024-03-26T02:01:49.418798+0000","last_undegraded":"2024-03-26T02:01:49.418798+0000","last_fullsized":"2024-03-26T02:01:49.418798+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:29:11.018800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.534404+0000","last_change":"2024-03-26T02:01:12.247847+0000","last_active":"2024-03-26T02:01:49.534404+0000","last_peered":"2024-03-26T02:01:49.534404+0000","last_clean":"2024-03-26T02:01:49.534404+0000","last_became_active":"2024-03-26T02:01:12.247522+0000","last_became_peered":"2024-03-26T02:01:12.247522+0000","last_unstale":"2024-03-26T02:01:49.534404+0000","last_undegraded":"2024-03-26T02:01:49.534404+0000","last_fullsized":"2024-03-26T02:01:49.534404+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:07:11.520915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419443+0000","last_change":"2024-03-26T02:01:08.209092+0000","last_active":"2024-03-26T02:01:49.419443+0000","last_peered":"2024-03-26T02:01:49.419443+0000","last_clean":"2024-03-26T02:01:49.419443+0000","last_became_active":"2024-03-26T02:01:08.207895+0000","last_became_peered":"2024-03-26T02:01:08.207895+0000","last_unstale":"2024-03-26T02:01:49.419443+0000","last_undegraded":"2024-03-26T02:01:49.419443+0000","last_fullsized":"2024-03-26T02:01:49.419443+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:50:14.560374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"67'17","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.420483+0000","last_change":"2024-03-26T02:01:10.226787+0000","last_active":"2024-03-26T02:01:49.420483+0000","last_peered":"2024-03-26T02:01:49.420483+0000","last_clean":"2024-03-26T02:01:49.420483+0000","last_became_active":"2024-03-26T02:01:10.226092+0000","last_became_peered":"2024-03-26T02:01:10.226092+0000","last_unstale":"2024-03-26T02:01:49.420483+0000","last_undegraded":"2024-03-26T02:01:49.420483+0000","last_fullsized":"2024-03-26T02:01:49.420483+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:05:30.553189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.422431+0000","last_change":"2024-03-26T02:01:14.253413+0000","last_active":"2024-03-26T02:01:49.422431+0000","last_peered":"2024-03-26T02:01:49.422431+0000","last_clean":"2024-03-26T02:01:49.422431+0000","last_became_active":"2024-03-26T02:01:14.253115+0000","last_became_peered":"2024-03-26T02:01:14.253115+0000","last_unstale":"2024-03-26T02:01:49.422431+0000","last_undegraded":"2024-03-26T02:01:49.422431+0000","last_fullsized":"2024-03-26T02:01:49.422431+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:31:39.354661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739519+0000","last_change":"2024-03-26T02:01:12.241559+0000","last_active":"2024-03-26T02:01:49.739519+0000","last_peered":"2024-03-26T02:01:49.739519+0000","last_clean":"2024-03-26T02:01:49.739519+0000","last_became_active":"2024-03-26T02:01:12.241211+0000","last_became_peered":"2024-03-26T02:01:12.241211+0000","last_unstale":"2024-03-26T02:01:49.739519+0000","last_undegraded":"2024-03-26T02:01:49.739519+0000","last_fullsized":"2024-03-26T02:01:49.739519+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:56:54.174990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":32,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.594551+0000","last_change":"2024-03-26T02:01:08.215824+0000","last_active":"2024-03-26T02:01:49.594551+0000","last_peered":"2024-03-26T02:01:49.594551+0000","last_clean":"2024-03-26T02:01:49.594551+0000","last_became_active":"2024-03-26T02:01:08.215539+0000","last_became_peered":"2024-03-26T02:01:08.215539+0000","last_unstale":"2024-03-26T02:01:49.594551+0000","last_undegraded":"2024-03-26T02:01:49.594551+0000","last_fullsized":"2024-03-26T02:01:49.594551+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:17:49.285575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":81,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.421142+0000","last_change":"2024-03-26T02:01:29.159078+0000","last_active":"2024-03-26T02:01:49.421142+0000","last_peered":"2024-03-26T02:01:49.421142+0000","last_clean":"2024-03-26T02:01:49.421142+0000","last_became_active":"2024-03-26T02:01:29.150851+0000","last_became_peered":"2024-03-26T02:01:29.150851+0000","last_unstale":"2024-03-26T02:01:49.421142+0000","last_undegraded":"2024-03-26T02:01:49.421142+0000","last_fullsized":"2024-03-26T02:01:49.421142+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:07:25.101349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.420640+0000","last_change":"2024-03-26T02:01:14.249417+0000","last_active":"2024-03-26T02:01:49.420640+0000","last_peered":"2024-03-26T02:01:49.420640+0000","last_clean":"2024-03-26T02:01:49.420640+0000","last_became_active":"2024-03-26T02:01:14.249132+0000","last_became_peered":"2024-03-26T02:01:14.249132+0000","last_unstale":"2024-03-26T02:01:49.420640+0000","last_undegraded":"2024-03-26T02:01:49.420640+0000","last_fullsized":"2024-03-26T02:01:49.420640+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:20:11.616069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":49,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:02:18.262419+0000","last_change":"2024-03-26T02:01:12.243689+0000","last_active":"2024-03-26T02:02:18.262419+0000","last_peered":"2024-03-26T02:02:18.262419+0000","last_clean":"2024-03-26T02:02:18.262419+0000","last_became_active":"2024-03-26T02:01:12.242674+0000","last_became_peered":"2024-03-26T02:01:12.242674+0000","last_unstale":"2024-03-26T02:02:18.262419+0000","last_undegraded":"2024-03-26T02:02:18.262419+0000","last_fullsized":"2024-03-26T02:02:18.262419+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:38:39.771989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419688+0000","last_change":"2024-03-26T02:01:08.209891+0000","last_active":"2024-03-26T02:01:49.419688+0000","last_peered":"2024-03-26T02:01:49.419688+0000","last_clean":"2024-03-26T02:01:49.419688+0000","last_became_active":"2024-03-26T02:01:08.209596+0000","last_became_peered":"2024-03-26T02:01:08.209596+0000","last_unstale":"2024-03-26T02:01:49.419688+0000","last_undegraded":"2024-03-26T02:01:49.419688+0000","last_fullsized":"2024-03-26T02:01:49.419688+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:20:03.580395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.420126+0000","last_change":"2024-03-26T02:01:10.226753+0000","last_active":"2024-03-26T02:01:49.420126+0000","last_peered":"2024-03-26T02:01:49.420126+0000","last_clean":"2024-03-26T02:01:49.420126+0000","last_became_active":"2024-03-26T02:01:10.224630+0000","last_became_peered":"2024-03-26T02:01:10.224630+0000","last_unstale":"2024-03-26T02:01:49.420126+0000","last_undegraded":"2024-03-26T02:01:49.420126+0000","last_fullsized":"2024-03-26T02:01:49.420126+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:06:50.698862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":590,"ondisk_log_size":590,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":66,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234004,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1302528,"data_stored":868050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495640,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28272,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716880,"statfs":{"total":95995035648,"available":95966085120,"internally_reserved":0,"allocated":1286144,"data_stored":867011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31900,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713252,"statfs":{"total":95995035648,"available":95962370048,"internally_reserved":0,"allocated":737280,"data_stored":277687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32460,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712692,"statfs":{"total":95995035648,"available":95961796608,"internally_reserved":0,"allocated":1310720,"data_stored":868282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31852,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713300,"statfs":{"total":95995035648,"available":95962419200,"internally_reserved":0,"allocated":688128,"data_stored":276567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182441,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31892,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713260,"statfs":{"total":95995035648,"available":95962378240,"internally_reserved":0,"allocated":729088,"data_stored":278011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018916,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27720,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":716800,"data_stored":277712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757278,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":696320,"data_stored":277066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T02:02:30.256 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T02:02:30.515 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:02:30.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:02:30 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:02:31.203 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:31 smithi072 ceph-mon[30177]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:02:31.203 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:31 smithi072 ceph-mon[30177]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:31.203 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:31 smithi072 ceph-mon[41619]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:02:31.203 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:31 smithi072 ceph-mon[41619]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:31.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:31 smithi173 ceph-mon[33839]: from='client.14721 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:02:31.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:31 smithi173 ceph-mon[33839]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:32.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:32 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:02:32.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:32 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:02:32.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:32 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:02:33.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.235+0000 7fb30dece700 1 -- 172.21.15.72:0/602932921 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb308101920 msgr2=0x7fb30810de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:33.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.235+0000 7fb30dece700 1 --2- 172.21.15.72:0/602932921 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb308101920 0x7fb30810de00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb30000b0a0 tx=0x7fb30000b3b0 comp rx=0 tx=0).stop 2024-03-26T02:02:33.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.235+0000 7fb30dece700 1 -- 172.21.15.72:0/602932921 shutdown_connections 2024-03-26T02:02:33.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.235+0000 7fb30dece700 1 --2- 172.21.15.72:0/602932921 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb308101920 0x7fb30810de00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:33.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.235+0000 7fb30dece700 1 --2- 172.21.15.72:0/602932921 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb308106fc0 0x7fb3081073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:33.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.235+0000 7fb30dece700 1 --2- 172.21.15.72:0/602932921 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb308100f80 0x7fb3081013e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:33.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.235+0000 7fb30dece700 1 -- 172.21.15.72:0/602932921 >> 172.21.15.72:0/602932921 conn(0x7fb308077eb0 msgr2=0x7fb3080782c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:33.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.235+0000 7fb30dece700 1 -- 172.21.15.72:0/602932921 shutdown_connections 2024-03-26T02:02:33.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.235+0000 7fb30dece700 1 -- 172.21.15.72:0/602932921 wait complete. 2024-03-26T02:02:33.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.236+0000 7fb30dece700 1 Processor -- start 2024-03-26T02:02:33.239 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.236+0000 7fb30dece700 1 -- start start 2024-03-26T02:02:33.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.236+0000 7fb30dece700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb308100f80 0x7fb308196da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:33.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.236+0000 7fb30dece700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb308101920 0x7fb3081972e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:33.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.236+0000 7fb30dece700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb308106fc0 0x7fb30819b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:33.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.236+0000 7fb30dece700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb30819bc80 con 0x7fb308100f80 2024-03-26T02:02:33.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.236+0000 7fb30dece700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb30819bdf0 con 0x7fb308106fc0 2024-03-26T02:02:33.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.236+0000 7fb30dece700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb30819bfb0 con 0x7fb308101920 2024-03-26T02:02:33.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.236+0000 7fb306ffd700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb308101920 0x7fb3081972e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:33.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.236+0000 7fb3077fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb308100f80 0x7fb308196da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:33.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.236+0000 7fb306ffd700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb308101920 0x7fb3081972e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:59114/0 (socket says 172.21.15.72:59114) 2024-03-26T02:02:33.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.236+0000 7fb306ffd700 1 -- 172.21.15.72:0/2649457140 learned_addr learned my addr 172.21.15.72:0/2649457140 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:02:33.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.237+0000 7fb307fff700 1 --2- 172.21.15.72:0/2649457140 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb308106fc0 0x7fb30819b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:33.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.237+0000 7fb306ffd700 1 -- 172.21.15.72:0/2649457140 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb308106fc0 msgr2=0x7fb30819b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:33.240 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.237+0000 7fb306ffd700 1 --2- 172.21.15.72:0/2649457140 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb308106fc0 0x7fb30819b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:33.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.237+0000 7fb306ffd700 1 -- 172.21.15.72:0/2649457140 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb308100f80 msgr2=0x7fb308196da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:33.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.237+0000 7fb306ffd700 1 --2- 172.21.15.72:0/2649457140 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb308100f80 0x7fb308196da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:33.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.237+0000 7fb306ffd700 1 -- 172.21.15.72:0/2649457140 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2fc009520 con 0x7fb308101920 2024-03-26T02:02:33.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.237+0000 7fb3077fe700 1 --2- 172.21.15.72:0/2649457140 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb308100f80 0x7fb308196da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:02:33.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.237+0000 7fb306ffd700 1 --2- 172.21.15.72:0/2649457140 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb308101920 0x7fb3081972e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc00eb30 tx=0x7fb2fc00ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:33.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.237+0000 7fb304ff9700 1 -- 172.21.15.72:0/2649457140 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2fc009880 con 0x7fb308101920 2024-03-26T02:02:33.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.237+0000 7fb30dece700 1 -- 172.21.15.72:0/2649457140 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb300009d00 con 0x7fb308101920 2024-03-26T02:02:33.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.237+0000 7fb304ff9700 1 -- 172.21.15.72:0/2649457140 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fb2fc0044d0 con 0x7fb308101920 2024-03-26T02:02:33.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.237+0000 7fb304ff9700 1 -- 172.21.15.72:0/2649457140 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2fc003e20 con 0x7fb308101920 2024-03-26T02:02:33.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.237+0000 7fb30dece700 1 -- 172.21.15.72:0/2649457140 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3081a3920 con 0x7fb308101920 2024-03-26T02:02:33.241 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.238+0000 7fb30dece700 1 -- 172.21.15.72:0/2649457140 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3081094e0 con 0x7fb308101920 2024-03-26T02:02:33.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.239+0000 7fb304ff9700 1 -- 172.21.15.72:0/2649457140 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7fb2fc003f80 con 0x7fb308101920 2024-03-26T02:02:33.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.239+0000 7fb304ff9700 1 --2- 172.21.15.72:0/2649457140 >> v2:172.21.15.72:6800/3246690770 conn(0x7fb2f006da80 0x7fb2f006ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:33.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.239+0000 7fb304ff9700 1 -- 172.21.15.72:0/2649457140 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6259+0+0 (secure 0 0 0) 0x7fb2fc014070 con 0x7fb308101920 2024-03-26T02:02:33.242 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.239+0000 7fb3077fe700 1 --2- 172.21.15.72:0/2649457140 >> v2:172.21.15.72:6800/3246690770 conn(0x7fb2f006da80 0x7fb2f006ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:33.243 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.239+0000 7fb3077fe700 1 --2- 172.21.15.72:0/2649457140 >> v2:172.21.15.72:6800/3246690770 conn(0x7fb2f006da80 0x7fb2f006ff40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8009de0 tx=0x7fb2f8009190 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:33.244 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.244+0000 7fb304ff9700 1 -- 172.21.15.72:0/2649457140 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fb2fc05abd0 con 0x7fb308101920 2024-03-26T02:02:33.449 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.448+0000 7fb30dece700 1 -- 172.21.15.72:0/2649457140 --> v2:172.21.15.72:6800/3246690770 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb308198100 con 0x7fb2f006da80 2024-03-26T02:02:33.449 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:33 smithi072 ceph-mon[30177]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:33.450 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:33 smithi072 ceph-mon[41619]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:33.458 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.458+0000 7fb304ff9700 1 -- 172.21.15.72:0/2649457140 <== mgr.24467 v2:172.21.15.72:6800/3246690770 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+346503 (secure 0 0 0) 0x7fb308198100 con 0x7fb2f006da80 2024-03-26T02:02:33.463 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T02:02:33.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.462+0000 7fb30dece700 1 -- 172.21.15.72:0/2649457140 >> v2:172.21.15.72:6800/3246690770 conn(0x7fb2f006da80 msgr2=0x7fb2f006ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:33.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.462+0000 7fb30dece700 1 --2- 172.21.15.72:0/2649457140 >> v2:172.21.15.72:6800/3246690770 conn(0x7fb2f006da80 0x7fb2f006ff40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8009de0 tx=0x7fb2f8009190 comp rx=0 tx=0).stop 2024-03-26T02:02:33.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.462+0000 7fb30dece700 1 -- 172.21.15.72:0/2649457140 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb308101920 msgr2=0x7fb3081972e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:33.465 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.462+0000 7fb30dece700 1 --2- 172.21.15.72:0/2649457140 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb308101920 0x7fb3081972e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc00eb30 tx=0x7fb2fc00ee40 comp rx=0 tx=0).stop 2024-03-26T02:02:33.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.462+0000 7fb30dece700 1 -- 172.21.15.72:0/2649457140 shutdown_connections 2024-03-26T02:02:33.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.462+0000 7fb30dece700 1 --2- 172.21.15.72:0/2649457140 >> v2:172.21.15.72:6800/3246690770 conn(0x7fb2f006da80 0x7fb2f006ff40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:33.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.462+0000 7fb30dece700 1 --2- 172.21.15.72:0/2649457140 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb308106fc0 0x7fb30819b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:33.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.462+0000 7fb30dece700 1 --2- 172.21.15.72:0/2649457140 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fb308100f80 0x7fb308196da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:33.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.462+0000 7fb30dece700 1 --2- 172.21.15.72:0/2649457140 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fb308101920 0x7fb3081972e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:33.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.462+0000 7fb30dece700 1 -- 172.21.15.72:0/2649457140 >> 172.21.15.72:0/2649457140 conn(0x7fb308077eb0 msgr2=0x7fb3080ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:33.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.463+0000 7fb30dece700 1 -- 172.21.15.72:0/2649457140 shutdown_connections 2024-03-26T02:02:33.466 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:33.463+0000 7fb30dece700 1 -- 172.21.15.72:0/2649457140 wait complete. 2024-03-26T02:02:33.466 INFO:teuthology.orchestra.run.smithi072.stderr:dumped all 2024-03-26T02:02:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:33 smithi173 ceph-mon[33839]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:34.458 INFO:teuthology.orchestra.run.smithi072.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2024-03-26T02:02:33.438931+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1298,"num_read_kb":1112,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":797,"ondisk_log_size":797,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":240088,"kb_used_data":7292,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":749721128,"statfs":{"total":767960285184,"available":767714435072,"internally_reserved":0,"allocated":7467008,"data_stored":3990386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002923"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419453+0000","last_change":"2024-03-26T02:01:14.254100+0000","last_active":"2024-03-26T02:01:49.419453+0000","last_peered":"2024-03-26T02:01:49.419453+0000","last_clean":"2024-03-26T02:01:49.419453+0000","last_became_active":"2024-03-26T02:01:14.253800+0000","last_became_peered":"2024-03-26T02:01:14.253800+0000","last_unstale":"2024-03-26T02:01:49.419453+0000","last_undegraded":"2024-03-26T02:01:49.419453+0000","last_fullsized":"2024-03-26T02:01:49.419453+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:32:05.985959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.738640+0000","last_change":"2024-03-26T02:01:12.244272+0000","last_active":"2024-03-26T02:01:49.738640+0000","last_peered":"2024-03-26T02:01:49.738640+0000","last_clean":"2024-03-26T02:01:49.738640+0000","last_became_active":"2024-03-26T02:01:12.243977+0000","last_became_peered":"2024-03-26T02:01:12.243977+0000","last_unstale":"2024-03-26T02:01:49.738640+0000","last_undegraded":"2024-03-26T02:01:49.738640+0000","last_fullsized":"2024-03-26T02:01:49.738640+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:47:52.673808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739283+0000","last_change":"2024-03-26T02:01:08.213024+0000","last_active":"2024-03-26T02:01:49.739283+0000","last_peered":"2024-03-26T02:01:49.739283+0000","last_clean":"2024-03-26T02:01:49.739283+0000","last_became_active":"2024-03-26T02:01:08.212743+0000","last_became_peered":"2024-03-26T02:01:08.212743+0000","last_unstale":"2024-03-26T02:01:49.739283+0000","last_undegraded":"2024-03-26T02:01:49.739283+0000","last_fullsized":"2024-03-26T02:01:49.739283+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:48:36.119186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"67'15","reported_seq":57,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419365+0000","last_change":"2024-03-26T02:01:10.224453+0000","last_active":"2024-03-26T02:01:49.419365+0000","last_peered":"2024-03-26T02:01:49.419365+0000","last_clean":"2024-03-26T02:01:49.419365+0000","last_became_active":"2024-03-26T02:01:10.223780+0000","last_became_peered":"2024-03-26T02:01:10.223780+0000","last_unstale":"2024-03-26T02:01:49.419365+0000","last_undegraded":"2024-03-26T02:01:49.419365+0000","last_fullsized":"2024-03-26T02:01:49.419365+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:08:07.751617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419502+0000","last_change":"2024-03-26T02:01:14.949106+0000","last_active":"2024-03-26T02:01:49.419502+0000","last_peered":"2024-03-26T02:01:49.419502+0000","last_clean":"2024-03-26T02:01:49.419502+0000","last_became_active":"2024-03-26T02:01:14.948745+0000","last_became_peered":"2024-03-26T02:01:14.948745+0000","last_unstale":"2024-03-26T02:01:49.419502+0000","last_undegraded":"2024-03-26T02:01:49.419502+0000","last_fullsized":"2024-03-26T02:01:49.419502+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:50:07.943908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419368+0000","last_change":"2024-03-26T02:01:12.241704+0000","last_active":"2024-03-26T02:01:49.419368+0000","last_peered":"2024-03-26T02:01:49.419368+0000","last_clean":"2024-03-26T02:01:49.419368+0000","last_became_active":"2024-03-26T02:01:12.240025+0000","last_became_peered":"2024-03-26T02:01:12.240025+0000","last_unstale":"2024-03-26T02:01:49.419368+0000","last_undegraded":"2024-03-26T02:01:49.419368+0000","last_fullsized":"2024-03-26T02:01:49.419368+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:37:28.701177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419631+0000","last_change":"2024-03-26T02:01:08.206463+0000","last_active":"2024-03-26T02:01:49.419631+0000","last_peered":"2024-03-26T02:01:49.419631+0000","last_clean":"2024-03-26T02:01:49.419631+0000","last_became_active":"2024-03-26T02:01:08.205735+0000","last_became_peered":"2024-03-26T02:01:08.205735+0000","last_unstale":"2024-03-26T02:01:49.419631+0000","last_undegraded":"2024-03-26T02:01:49.419631+0000","last_fullsized":"2024-03-26T02:01:49.419631+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:04:31.024748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"67'39","reported_seq":102,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.418742+0000","last_change":"2024-03-26T02:01:10.220412+0000","last_active":"2024-03-26T02:01:49.418742+0000","last_peered":"2024-03-26T02:01:49.418742+0000","last_clean":"2024-03-26T02:01:49.418742+0000","last_became_active":"2024-03-26T02:01:10.220109+0000","last_became_peered":"2024-03-26T02:01:10.220109+0000","last_unstale":"2024-03-26T02:01:49.418742+0000","last_undegraded":"2024-03-26T02:01:49.418742+0000","last_fullsized":"2024-03-26T02:01:49.418742+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:53:46.904838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.737817+0000","last_change":"2024-03-26T02:01:14.250830+0000","last_active":"2024-03-26T02:01:49.737817+0000","last_peered":"2024-03-26T02:01:49.737817+0000","last_clean":"2024-03-26T02:01:49.737817+0000","last_became_active":"2024-03-26T02:01:14.250492+0000","last_became_peered":"2024-03-26T02:01:14.250492+0000","last_unstale":"2024-03-26T02:01:49.737817+0000","last_undegraded":"2024-03-26T02:01:49.737817+0000","last_fullsized":"2024-03-26T02:01:49.737817+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:33:15.249424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.533755+0000","last_change":"2024-03-26T02:01:12.242865+0000","last_active":"2024-03-26T02:01:49.533755+0000","last_peered":"2024-03-26T02:01:49.533755+0000","last_clean":"2024-03-26T02:01:49.533755+0000","last_became_active":"2024-03-26T02:01:12.242552+0000","last_became_peered":"2024-03-26T02:01:12.242552+0000","last_unstale":"2024-03-26T02:01:49.533755+0000","last_undegraded":"2024-03-26T02:01:49.533755+0000","last_fullsized":"2024-03-26T02:01:49.533755+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:07:43.897667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.533976+0000","last_change":"2024-03-26T02:01:08.214349+0000","last_active":"2024-03-26T02:01:49.533976+0000","last_peered":"2024-03-26T02:01:49.533976+0000","last_clean":"2024-03-26T02:01:49.533976+0000","last_became_active":"2024-03-26T02:01:08.214041+0000","last_became_peered":"2024-03-26T02:01:08.214041+0000","last_unstale":"2024-03-26T02:01:49.533976+0000","last_undegraded":"2024-03-26T02:01:49.533976+0000","last_fullsized":"2024-03-26T02:01:49.533976+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:25:28.902727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"67'21","reported_seq":69,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419730+0000","last_change":"2024-03-26T02:01:10.222145+0000","last_active":"2024-03-26T02:01:49.419730+0000","last_peered":"2024-03-26T02:01:49.419730+0000","last_clean":"2024-03-26T02:01:49.419730+0000","last_became_active":"2024-03-26T02:01:10.221725+0000","last_became_peered":"2024-03-26T02:01:10.221725+0000","last_unstale":"2024-03-26T02:01:49.419730+0000","last_undegraded":"2024-03-26T02:01:49.419730+0000","last_fullsized":"2024-03-26T02:01:49.419730+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:30:14.673620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.533437+0000","last_change":"2024-03-26T02:01:14.254015+0000","last_active":"2024-03-26T02:01:49.533437+0000","last_peered":"2024-03-26T02:01:49.533437+0000","last_clean":"2024-03-26T02:01:49.533437+0000","last_became_active":"2024-03-26T02:01:14.253698+0000","last_became_peered":"2024-03-26T02:01:14.253698+0000","last_unstale":"2024-03-26T02:01:49.533437+0000","last_undegraded":"2024-03-26T02:01:49.533437+0000","last_fullsized":"2024-03-26T02:01:49.533437+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:27:07.758079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.427713+0000","last_change":"2024-03-26T02:01:12.247571+0000","last_active":"2024-03-26T02:01:49.427713+0000","last_peered":"2024-03-26T02:01:49.427713+0000","last_clean":"2024-03-26T02:01:49.427713+0000","last_became_active":"2024-03-26T02:01:12.247293+0000","last_became_peered":"2024-03-26T02:01:12.247293+0000","last_unstale":"2024-03-26T02:01:49.427713+0000","last_undegraded":"2024-03-26T02:01:49.427713+0000","last_fullsized":"2024-03-26T02:01:49.427713+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:36:47.223790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.533888+0000","last_change":"2024-03-26T02:01:08.211815+0000","last_active":"2024-03-26T02:01:49.533888+0000","last_peered":"2024-03-26T02:01:49.533888+0000","last_clean":"2024-03-26T02:01:49.533888+0000","last_became_active":"2024-03-26T02:01:08.211498+0000","last_became_peered":"2024-03-26T02:01:08.211498+0000","last_unstale":"2024-03-26T02:01:49.533888+0000","last_undegraded":"2024-03-26T02:01:49.533888+0000","last_fullsized":"2024-03-26T02:01:49.533888+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:41:29.582259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"67'11","reported_seq":51,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419974+0000","last_change":"2024-03-26T02:01:10.227169+0000","last_active":"2024-03-26T02:01:49.419974+0000","last_peered":"2024-03-26T02:01:49.419974+0000","last_clean":"2024-03-26T02:01:49.419974+0000","last_became_active":"2024-03-26T02:01:10.226812+0000","last_became_peered":"2024-03-26T02:01:10.226812+0000","last_unstale":"2024-03-26T02:01:49.419974+0000","last_undegraded":"2024-03-26T02:01:49.419974+0000","last_fullsized":"2024-03-26T02:01:49.419974+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:46:25.654115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.420522+0000","last_change":"2024-03-26T02:01:14.947318+0000","last_active":"2024-03-26T02:01:49.420522+0000","last_peered":"2024-03-26T02:01:49.420522+0000","last_clean":"2024-03-26T02:01:49.420522+0000","last_became_active":"2024-03-26T02:01:14.947010+0000","last_became_peered":"2024-03-26T02:01:14.947010+0000","last_unstale":"2024-03-26T02:01:49.420522+0000","last_undegraded":"2024-03-26T02:01:49.420522+0000","last_fullsized":"2024-03-26T02:01:49.420522+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:21:01.716389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.738374+0000","last_change":"2024-03-26T02:01:12.243576+0000","last_active":"2024-03-26T02:01:49.738374+0000","last_peered":"2024-03-26T02:01:49.738374+0000","last_clean":"2024-03-26T02:01:49.738374+0000","last_became_active":"2024-03-26T02:01:12.243288+0000","last_became_peered":"2024-03-26T02:01:12.243288+0000","last_unstale":"2024-03-26T02:01:49.738374+0000","last_undegraded":"2024-03-26T02:01:49.738374+0000","last_fullsized":"2024-03-26T02:01:49.738374+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:57:53.502381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739534+0000","last_change":"2024-03-26T02:01:08.212432+0000","last_active":"2024-03-26T02:01:49.739534+0000","last_peered":"2024-03-26T02:01:49.739534+0000","last_clean":"2024-03-26T02:01:49.739534+0000","last_became_active":"2024-03-26T02:01:08.211840+0000","last_became_peered":"2024-03-26T02:01:08.211840+0000","last_unstale":"2024-03-26T02:01:49.739534+0000","last_undegraded":"2024-03-26T02:01:49.739534+0000","last_fullsized":"2024-03-26T02:01:49.739534+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:22:47.051270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.422035+0000","last_change":"2024-03-26T02:01:10.223652+0000","last_active":"2024-03-26T02:01:49.422035+0000","last_peered":"2024-03-26T02:01:49.422035+0000","last_clean":"2024-03-26T02:01:49.422035+0000","last_became_active":"2024-03-26T02:01:10.223283+0000","last_became_peered":"2024-03-26T02:01:10.223283+0000","last_unstale":"2024-03-26T02:01:49.422035+0000","last_undegraded":"2024-03-26T02:01:49.422035+0000","last_fullsized":"2024-03-26T02:01:49.422035+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:15:56.314001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419792+0000","last_change":"2024-03-26T02:01:14.947449+0000","last_active":"2024-03-26T02:01:49.419792+0000","last_peered":"2024-03-26T02:01:49.419792+0000","last_clean":"2024-03-26T02:01:49.419792+0000","last_became_active":"2024-03-26T02:01:14.947074+0000","last_became_peered":"2024-03-26T02:01:14.947074+0000","last_unstale":"2024-03-26T02:01:49.419792+0000","last_undegraded":"2024-03-26T02:01:49.419792+0000","last_fullsized":"2024-03-26T02:01:49.419792+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:05:57.273489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.420106+0000","last_change":"2024-03-26T02:01:12.237378+0000","last_active":"2024-03-26T02:01:49.420106+0000","last_peered":"2024-03-26T02:01:49.420106+0000","last_clean":"2024-03-26T02:01:49.420106+0000","last_became_active":"2024-03-26T02:01:12.237085+0000","last_became_peered":"2024-03-26T02:01:12.237085+0000","last_unstale":"2024-03-26T02:01:49.420106+0000","last_undegraded":"2024-03-26T02:01:49.420106+0000","last_fullsized":"2024-03-26T02:01:49.420106+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:58:32.416981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419745+0000","last_change":"2024-03-26T02:01:08.207724+0000","last_active":"2024-03-26T02:01:49.419745+0000","last_peered":"2024-03-26T02:01:49.419745+0000","last_clean":"2024-03-26T02:01:49.419745+0000","last_became_active":"2024-03-26T02:01:08.207198+0000","last_became_peered":"2024-03-26T02:01:08.207198+0000","last_unstale":"2024-03-26T02:01:49.419745+0000","last_undegraded":"2024-03-26T02:01:49.419745+0000","last_fullsized":"2024-03-26T02:01:49.419745+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:32:53.486843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"67'18","reported_seq":64,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419090+0000","last_change":"2024-03-26T02:01:10.226650+0000","last_active":"2024-03-26T02:01:49.419090+0000","last_peered":"2024-03-26T02:01:49.419090+0000","last_clean":"2024-03-26T02:01:49.419090+0000","last_became_active":"2024-03-26T02:01:10.224616+0000","last_became_peered":"2024-03-26T02:01:10.224616+0000","last_unstale":"2024-03-26T02:01:49.419090+0000","last_undegraded":"2024-03-26T02:01:49.419090+0000","last_fullsized":"2024-03-26T02:01:49.419090+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:05:28.121022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"67'1","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.427911+0000","last_change":"2024-03-26T02:01:14.250949+0000","last_active":"2024-03-26T02:01:49.427911+0000","last_peered":"2024-03-26T02:01:49.427911+0000","last_clean":"2024-03-26T02:01:49.427911+0000","last_became_active":"2024-03-26T02:01:14.250647+0000","last_became_peered":"2024-03-26T02:01:14.250647+0000","last_unstale":"2024-03-26T02:01:49.427911+0000","last_undegraded":"2024-03-26T02:01:49.427911+0000","last_fullsized":"2024-03-26T02:01:49.427911+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:55:59.190070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"67'8","reported_seq":55,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:02:23.263664+0000","last_change":"2024-03-26T02:01:12.244114+0000","last_active":"2024-03-26T02:02:23.263664+0000","last_peered":"2024-03-26T02:02:23.263664+0000","last_clean":"2024-03-26T02:02:23.263664+0000","last_became_active":"2024-03-26T02:01:12.243732+0000","last_became_peered":"2024-03-26T02:01:12.243732+0000","last_unstale":"2024-03-26T02:02:23.263664+0000","last_undegraded":"2024-03-26T02:02:23.263664+0000","last_fullsized":"2024-03-26T02:02:23.263664+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:41:52.651373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"60'1","reported_seq":34,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.738673+0000","last_change":"2024-03-26T02:01:08.212504+0000","last_active":"2024-03-26T02:01:49.738673+0000","last_peered":"2024-03-26T02:01:49.738673+0000","last_clean":"2024-03-26T02:01:49.738673+0000","last_became_active":"2024-03-26T02:01:08.212175+0000","last_became_peered":"2024-03-26T02:01:08.212175+0000","last_unstale":"2024-03-26T02:01:49.738673+0000","last_undegraded":"2024-03-26T02:01:49.738673+0000","last_fullsized":"2024-03-26T02:01:49.738673+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:20:33.512281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"67'16","reported_seq":56,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739660+0000","last_change":"2024-03-26T02:01:10.220706+0000","last_active":"2024-03-26T02:01:49.739660+0000","last_peered":"2024-03-26T02:01:49.739660+0000","last_clean":"2024-03-26T02:01:49.739660+0000","last_became_active":"2024-03-26T02:01:10.220421+0000","last_became_peered":"2024-03-26T02:01:10.220421+0000","last_unstale":"2024-03-26T02:01:49.739660+0000","last_undegraded":"2024-03-26T02:01:49.739660+0000","last_fullsized":"2024-03-26T02:01:49.739660+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:58:04.956823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.738066+0000","last_change":"2024-03-26T02:01:14.256824+0000","last_active":"2024-03-26T02:01:49.738066+0000","last_peered":"2024-03-26T02:01:49.738066+0000","last_clean":"2024-03-26T02:01:49.738066+0000","last_became_active":"2024-03-26T02:01:14.256483+0000","last_became_peered":"2024-03-26T02:01:14.256483+0000","last_unstale":"2024-03-26T02:01:49.738066+0000","last_undegraded":"2024-03-26T02:01:49.738066+0000","last_fullsized":"2024-03-26T02:01:49.738066+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:35:33.068064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739859+0000","last_change":"2024-03-26T02:01:12.247491+0000","last_active":"2024-03-26T02:01:49.739859+0000","last_peered":"2024-03-26T02:01:49.739859+0000","last_clean":"2024-03-26T02:01:49.739859+0000","last_became_active":"2024-03-26T02:01:12.247172+0000","last_became_peered":"2024-03-26T02:01:12.247172+0000","last_unstale":"2024-03-26T02:01:49.739859+0000","last_undegraded":"2024-03-26T02:01:49.739859+0000","last_fullsized":"2024-03-26T02:01:49.739859+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:24:12.549870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.418343+0000","last_change":"2024-03-26T02:01:08.209079+0000","last_active":"2024-03-26T02:01:49.418343+0000","last_peered":"2024-03-26T02:01:49.418343+0000","last_clean":"2024-03-26T02:01:49.418343+0000","last_became_active":"2024-03-26T02:01:08.207998+0000","last_became_peered":"2024-03-26T02:01:08.207998+0000","last_unstale":"2024-03-26T02:01:49.418343+0000","last_undegraded":"2024-03-26T02:01:49.418343+0000","last_fullsized":"2024-03-26T02:01:49.418343+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:16:59.366565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"67'17","reported_seq":59,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.595048+0000","last_change":"2024-03-26T02:01:10.221845+0000","last_active":"2024-03-26T02:01:49.595048+0000","last_peered":"2024-03-26T02:01:49.595048+0000","last_clean":"2024-03-26T02:01:49.595048+0000","last_became_active":"2024-03-26T02:01:10.221514+0000","last_became_peered":"2024-03-26T02:01:10.221514+0000","last_unstale":"2024-03-26T02:01:49.595048+0000","last_undegraded":"2024-03-26T02:01:49.595048+0000","last_fullsized":"2024-03-26T02:01:49.595048+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:20:23.001489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.427681+0000","last_change":"2024-03-26T02:01:14.256989+0000","last_active":"2024-03-26T02:01:49.427681+0000","last_peered":"2024-03-26T02:01:49.427681+0000","last_clean":"2024-03-26T02:01:49.427681+0000","last_became_active":"2024-03-26T02:01:14.256633+0000","last_became_peered":"2024-03-26T02:01:14.256633+0000","last_unstale":"2024-03-26T02:01:49.427681+0000","last_undegraded":"2024-03-26T02:01:49.427681+0000","last_fullsized":"2024-03-26T02:01:49.427681+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:55:24.654808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":24,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.594352+0000","last_change":"2024-03-26T02:01:12.242199+0000","last_active":"2024-03-26T02:01:49.594352+0000","last_peered":"2024-03-26T02:01:49.594352+0000","last_clean":"2024-03-26T02:01:49.594352+0000","last_became_active":"2024-03-26T02:01:12.241923+0000","last_became_peered":"2024-03-26T02:01:12.241923+0000","last_unstale":"2024-03-26T02:01:49.594352+0000","last_undegraded":"2024-03-26T02:01:49.594352+0000","last_fullsized":"2024-03-26T02:01:49.594352+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:55:48.612436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739198+0000","last_change":"2024-03-26T02:01:08.208640+0000","last_active":"2024-03-26T02:01:49.739198+0000","last_peered":"2024-03-26T02:01:49.739198+0000","last_clean":"2024-03-26T02:01:49.739198+0000","last_became_active":"2024-03-26T02:01:08.208365+0000","last_became_peered":"2024-03-26T02:01:08.208365+0000","last_unstale":"2024-03-26T02:01:49.739198+0000","last_undegraded":"2024-03-26T02:01:49.739198+0000","last_fullsized":"2024-03-26T02:01:49.739198+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:09:12.148252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"67'4","reported_seq":35,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.418425+0000","last_change":"2024-03-26T02:01:10.223832+0000","last_active":"2024-03-26T02:01:49.418425+0000","last_peered":"2024-03-26T02:01:49.418425+0000","last_clean":"2024-03-26T02:01:49.418425+0000","last_became_active":"2024-03-26T02:01:10.223496+0000","last_became_peered":"2024-03-26T02:01:10.223496+0000","last_unstale":"2024-03-26T02:01:49.418425+0000","last_undegraded":"2024-03-26T02:01:49.418425+0000","last_fullsized":"2024-03-26T02:01:49.418425+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:10:07.203212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"67'1","reported_seq":22,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419383+0000","last_change":"2024-03-26T02:01:14.949310+0000","last_active":"2024-03-26T02:01:49.419383+0000","last_peered":"2024-03-26T02:01:49.419383+0000","last_clean":"2024-03-26T02:01:49.419383+0000","last_became_active":"2024-03-26T02:01:14.948814+0000","last_became_peered":"2024-03-26T02:01:14.948814+0000","last_unstale":"2024-03-26T02:01:49.419383+0000","last_undegraded":"2024-03-26T02:01:49.419383+0000","last_fullsized":"2024-03-26T02:01:49.419383+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:43:59.999251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.427970+0000","last_change":"2024-03-26T02:01:12.245655+0000","last_active":"2024-03-26T02:01:49.427970+0000","last_peered":"2024-03-26T02:01:49.427970+0000","last_clean":"2024-03-26T02:01:49.427970+0000","last_became_active":"2024-03-26T02:01:12.245388+0000","last_became_peered":"2024-03-26T02:01:12.245388+0000","last_unstale":"2024-03-26T02:01:49.427970+0000","last_undegraded":"2024-03-26T02:01:49.427970+0000","last_fullsized":"2024-03-26T02:01:49.427970+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:14:14.607316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.534195+0000","last_change":"2024-03-26T02:01:08.211997+0000","last_active":"2024-03-26T02:01:49.534195+0000","last_peered":"2024-03-26T02:01:49.534195+0000","last_clean":"2024-03-26T02:01:49.534195+0000","last_became_active":"2024-03-26T02:01:08.211528+0000","last_became_peered":"2024-03-26T02:01:08.211528+0000","last_unstale":"2024-03-26T02:01:49.534195+0000","last_undegraded":"2024-03-26T02:01:49.534195+0000","last_fullsized":"2024-03-26T02:01:49.534195+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:55:24.339095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"67'17","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.418680+0000","last_change":"2024-03-26T02:01:10.224403+0000","last_active":"2024-03-26T02:01:49.418680+0000","last_peered":"2024-03-26T02:01:49.418680+0000","last_clean":"2024-03-26T02:01:49.418680+0000","last_became_active":"2024-03-26T02:01:10.222828+0000","last_became_peered":"2024-03-26T02:01:10.222828+0000","last_unstale":"2024-03-26T02:01:49.418680+0000","last_undegraded":"2024-03-26T02:01:49.418680+0000","last_fullsized":"2024-03-26T02:01:49.418680+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:52:50.179227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.737634+0000","last_change":"2024-03-26T02:01:14.251801+0000","last_active":"2024-03-26T02:01:49.737634+0000","last_peered":"2024-03-26T02:01:49.737634+0000","last_clean":"2024-03-26T02:01:49.737634+0000","last_became_active":"2024-03-26T02:01:14.251481+0000","last_became_peered":"2024-03-26T02:01:14.251481+0000","last_unstale":"2024-03-26T02:01:49.737634+0000","last_undegraded":"2024-03-26T02:01:49.737634+0000","last_fullsized":"2024-03-26T02:01:49.737634+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:31:01.718454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419601+0000","last_change":"2024-03-26T02:01:12.242618+0000","last_active":"2024-03-26T02:01:49.419601+0000","last_peered":"2024-03-26T02:01:49.419601+0000","last_clean":"2024-03-26T02:01:49.419601+0000","last_became_active":"2024-03-26T02:01:12.242307+0000","last_became_peered":"2024-03-26T02:01:12.242307+0000","last_unstale":"2024-03-26T02:01:49.419601+0000","last_undegraded":"2024-03-26T02:01:49.419601+0000","last_fullsized":"2024-03-26T02:01:49.419601+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:28:38.852684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.428150+0000","last_change":"2024-03-26T02:01:08.214034+0000","last_active":"2024-03-26T02:01:49.428150+0000","last_peered":"2024-03-26T02:01:49.428150+0000","last_clean":"2024-03-26T02:01:49.428150+0000","last_became_active":"2024-03-26T02:01:08.213753+0000","last_became_peered":"2024-03-26T02:01:08.213753+0000","last_unstale":"2024-03-26T02:01:49.428150+0000","last_undegraded":"2024-03-26T02:01:49.428150+0000","last_fullsized":"2024-03-26T02:01:49.428150+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:33:54.305735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"67'9","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.738540+0000","last_change":"2024-03-26T02:01:10.224719+0000","last_active":"2024-03-26T02:01:49.738540+0000","last_peered":"2024-03-26T02:01:49.738540+0000","last_clean":"2024-03-26T02:01:49.738540+0000","last_became_active":"2024-03-26T02:01:10.224459+0000","last_became_peered":"2024-03-26T02:01:10.224459+0000","last_unstale":"2024-03-26T02:01:49.738540+0000","last_undegraded":"2024-03-26T02:01:49.738540+0000","last_fullsized":"2024-03-26T02:01:49.738540+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:13:50.902683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.418403+0000","last_change":"2024-03-26T02:01:14.255613+0000","last_active":"2024-03-26T02:01:49.418403+0000","last_peered":"2024-03-26T02:01:49.418403+0000","last_clean":"2024-03-26T02:01:49.418403+0000","last_became_active":"2024-03-26T02:01:14.255250+0000","last_became_peered":"2024-03-26T02:01:14.255250+0000","last_unstale":"2024-03-26T02:01:49.418403+0000","last_undegraded":"2024-03-26T02:01:49.418403+0000","last_fullsized":"2024-03-26T02:01:49.418403+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:25:12.743536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.738409+0000","last_change":"2024-03-26T02:01:12.241570+0000","last_active":"2024-03-26T02:01:49.738409+0000","last_peered":"2024-03-26T02:01:49.738409+0000","last_clean":"2024-03-26T02:01:49.738409+0000","last_became_active":"2024-03-26T02:01:12.241250+0000","last_became_peered":"2024-03-26T02:01:12.241250+0000","last_unstale":"2024-03-26T02:01:49.738409+0000","last_undegraded":"2024-03-26T02:01:49.738409+0000","last_fullsized":"2024-03-26T02:01:49.738409+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:06:35.375774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.420294+0000","last_change":"2024-03-26T02:01:08.207936+0000","last_active":"2024-03-26T02:01:49.420294+0000","last_peered":"2024-03-26T02:01:49.420294+0000","last_clean":"2024-03-26T02:01:49.420294+0000","last_became_active":"2024-03-26T02:01:08.207656+0000","last_became_peered":"2024-03-26T02:01:08.207656+0000","last_unstale":"2024-03-26T02:01:49.420294+0000","last_undegraded":"2024-03-26T02:01:49.420294+0000","last_fullsized":"2024-03-26T02:01:49.420294+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:14:26.153078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"67'11","reported_seq":48,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.420231+0000","last_change":"2024-03-26T02:01:10.226699+0000","last_active":"2024-03-26T02:01:49.420231+0000","last_peered":"2024-03-26T02:01:49.420231+0000","last_clean":"2024-03-26T02:01:49.420231+0000","last_became_active":"2024-03-26T02:01:10.226402+0000","last_became_peered":"2024-03-26T02:01:10.226402+0000","last_unstale":"2024-03-26T02:01:49.420231+0000","last_undegraded":"2024-03-26T02:01:49.420231+0000","last_fullsized":"2024-03-26T02:01:49.420231+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:47:27.617674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739385+0000","last_change":"2024-03-26T02:01:14.253866+0000","last_active":"2024-03-26T02:01:49.739385+0000","last_peered":"2024-03-26T02:01:49.739385+0000","last_clean":"2024-03-26T02:01:49.739385+0000","last_became_active":"2024-03-26T02:01:14.253565+0000","last_became_peered":"2024-03-26T02:01:14.253565+0000","last_unstale":"2024-03-26T02:01:49.739385+0000","last_undegraded":"2024-03-26T02:01:49.739385+0000","last_fullsized":"2024-03-26T02:01:49.739385+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:30:07.931245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:02:23.263544+0000","last_change":"2024-03-26T02:01:12.244489+0000","last_active":"2024-03-26T02:02:23.263544+0000","last_peered":"2024-03-26T02:02:23.263544+0000","last_clean":"2024-03-26T02:02:23.263544+0000","last_became_active":"2024-03-26T02:01:12.244214+0000","last_became_peered":"2024-03-26T02:01:12.244214+0000","last_unstale":"2024-03-26T02:02:23.263544+0000","last_undegraded":"2024-03-26T02:02:23.263544+0000","last_fullsized":"2024-03-26T02:02:23.263544+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:48:08.608031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.427747+0000","last_change":"2024-03-26T02:01:08.215223+0000","last_active":"2024-03-26T02:01:49.427747+0000","last_peered":"2024-03-26T02:01:49.427747+0000","last_clean":"2024-03-26T02:01:49.427747+0000","last_became_active":"2024-03-26T02:01:08.214934+0000","last_became_peered":"2024-03-26T02:01:08.214934+0000","last_unstale":"2024-03-26T02:01:49.427747+0000","last_undegraded":"2024-03-26T02:01:49.427747+0000","last_fullsized":"2024-03-26T02:01:49.427747+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:38:54.711852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"67'22","reported_seq":90,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419032+0000","last_change":"2024-03-26T02:01:28.905784+0000","last_active":"2024-03-26T02:01:49.419032+0000","last_peered":"2024-03-26T02:01:49.419032+0000","last_clean":"2024-03-26T02:01:49.419032+0000","last_became_active":"2024-03-26T02:01:28.897223+0000","last_became_peered":"2024-03-26T02:01:28.897223+0000","last_unstale":"2024-03-26T02:01:49.419032+0000","last_undegraded":"2024-03-26T02:01:49.419032+0000","last_fullsized":"2024-03-26T02:01:49.419032+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:58:57.814858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.420997+0000","last_change":"2024-03-26T02:01:14.251114+0000","last_active":"2024-03-26T02:01:49.420997+0000","last_peered":"2024-03-26T02:01:49.420997+0000","last_clean":"2024-03-26T02:01:49.420997+0000","last_became_active":"2024-03-26T02:01:14.250786+0000","last_became_peered":"2024-03-26T02:01:14.250786+0000","last_unstale":"2024-03-26T02:01:49.420997+0000","last_undegraded":"2024-03-26T02:01:49.420997+0000","last_fullsized":"2024-03-26T02:01:49.420997+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:11:03.970677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:02:23.263375+0000","last_change":"2024-03-26T02:01:12.238242+0000","last_active":"2024-03-26T02:02:23.263375+0000","last_peered":"2024-03-26T02:02:23.263375+0000","last_clean":"2024-03-26T02:02:23.263375+0000","last_became_active":"2024-03-26T02:01:12.237645+0000","last_became_peered":"2024-03-26T02:01:12.237645+0000","last_unstale":"2024-03-26T02:02:23.263375+0000","last_undegraded":"2024-03-26T02:02:23.263375+0000","last_fullsized":"2024-03-26T02:02:23.263375+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:03:39.764784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.738434+0000","last_change":"2024-03-26T02:01:08.209746+0000","last_active":"2024-03-26T02:01:49.738434+0000","last_peered":"2024-03-26T02:01:49.738434+0000","last_clean":"2024-03-26T02:01:49.738434+0000","last_became_active":"2024-03-26T02:01:08.209459+0000","last_became_peered":"2024-03-26T02:01:08.209459+0000","last_unstale":"2024-03-26T02:01:49.738434+0000","last_undegraded":"2024-03-26T02:01:49.738434+0000","last_fullsized":"2024-03-26T02:01:49.738434+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:21:34.155937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"67'15","reported_seq":57,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.534270+0000","last_change":"2024-03-26T02:01:10.223496+0000","last_active":"2024-03-26T02:01:49.534270+0000","last_peered":"2024-03-26T02:01:49.534270+0000","last_clean":"2024-03-26T02:01:49.534270+0000","last_became_active":"2024-03-26T02:01:10.223197+0000","last_became_peered":"2024-03-26T02:01:10.223197+0000","last_unstale":"2024-03-26T02:01:49.534270+0000","last_undegraded":"2024-03-26T02:01:49.534270+0000","last_fullsized":"2024-03-26T02:01:49.534270+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:46:50.071363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.422478+0000","last_change":"2024-03-26T02:01:14.250352+0000","last_active":"2024-03-26T02:01:49.422478+0000","last_peered":"2024-03-26T02:01:49.422478+0000","last_clean":"2024-03-26T02:01:49.422478+0000","last_became_active":"2024-03-26T02:01:14.250046+0000","last_became_peered":"2024-03-26T02:01:14.250046+0000","last_unstale":"2024-03-26T02:01:49.422478+0000","last_undegraded":"2024-03-26T02:01:49.422478+0000","last_fullsized":"2024-03-26T02:01:49.422478+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:13:25.433809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.420171+0000","last_change":"2024-03-26T02:01:12.241307+0000","last_active":"2024-03-26T02:01:49.420171+0000","last_peered":"2024-03-26T02:01:49.420171+0000","last_clean":"2024-03-26T02:01:49.420171+0000","last_became_active":"2024-03-26T02:01:12.241004+0000","last_became_peered":"2024-03-26T02:01:12.241004+0000","last_unstale":"2024-03-26T02:01:49.420171+0000","last_undegraded":"2024-03-26T02:01:49.420171+0000","last_fullsized":"2024-03-26T02:01:49.420171+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:42:28.224971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.427791+0000","last_change":"2024-03-26T02:01:08.213532+0000","last_active":"2024-03-26T02:01:49.427791+0000","last_peered":"2024-03-26T02:01:49.427791+0000","last_clean":"2024-03-26T02:01:49.427791+0000","last_became_active":"2024-03-26T02:01:08.212567+0000","last_became_peered":"2024-03-26T02:01:08.212567+0000","last_unstale":"2024-03-26T02:01:49.427791+0000","last_undegraded":"2024-03-26T02:01:49.427791+0000","last_fullsized":"2024-03-26T02:01:49.427791+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:46:06.440663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"67'23","reported_seq":71,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739777+0000","last_change":"2024-03-26T02:01:10.220378+0000","last_active":"2024-03-26T02:01:49.739777+0000","last_peered":"2024-03-26T02:01:49.739777+0000","last_clean":"2024-03-26T02:01:49.739777+0000","last_became_active":"2024-03-26T02:01:10.220089+0000","last_became_peered":"2024-03-26T02:01:10.220089+0000","last_unstale":"2024-03-26T02:01:49.739777+0000","last_undegraded":"2024-03-26T02:01:49.739777+0000","last_fullsized":"2024-03-26T02:01:49.739777+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:10:13.561692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.428146+0000","last_change":"2024-03-26T02:01:14.949065+0000","last_active":"2024-03-26T02:01:49.428146+0000","last_peered":"2024-03-26T02:01:49.428146+0000","last_clean":"2024-03-26T02:01:49.428146+0000","last_became_active":"2024-03-26T02:01:14.948757+0000","last_became_peered":"2024-03-26T02:01:14.948757+0000","last_unstale":"2024-03-26T02:01:49.428146+0000","last_undegraded":"2024-03-26T02:01:49.428146+0000","last_fullsized":"2024-03-26T02:01:49.428146+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:52:26.089697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.534200+0000","last_change":"2024-03-26T02:01:12.243768+0000","last_active":"2024-03-26T02:01:49.534200+0000","last_peered":"2024-03-26T02:01:49.534200+0000","last_clean":"2024-03-26T02:01:49.534200+0000","last_became_active":"2024-03-26T02:01:12.243495+0000","last_became_peered":"2024-03-26T02:01:12.243495+0000","last_unstale":"2024-03-26T02:01:49.534200+0000","last_undegraded":"2024-03-26T02:01:49.534200+0000","last_fullsized":"2024-03-26T02:01:49.534200+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:31:48.783807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":32,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.594280+0000","last_change":"2024-03-26T02:01:08.214330+0000","last_active":"2024-03-26T02:01:49.594280+0000","last_peered":"2024-03-26T02:01:49.594280+0000","last_clean":"2024-03-26T02:01:49.594280+0000","last_became_active":"2024-03-26T02:01:08.214032+0000","last_became_peered":"2024-03-26T02:01:08.214032+0000","last_unstale":"2024-03-26T02:01:49.594280+0000","last_undegraded":"2024-03-26T02:01:49.594280+0000","last_fullsized":"2024-03-26T02:01:49.594280+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:49:32.455174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"67'12","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.420254+0000","last_change":"2024-03-26T02:01:10.226283+0000","last_active":"2024-03-26T02:01:49.420254+0000","last_peered":"2024-03-26T02:01:49.420254+0000","last_clean":"2024-03-26T02:01:49.420254+0000","last_became_active":"2024-03-26T02:01:10.224813+0000","last_became_peered":"2024-03-26T02:01:10.224813+0000","last_unstale":"2024-03-26T02:01:49.420254+0000","last_undegraded":"2024-03-26T02:01:49.420254+0000","last_fullsized":"2024-03-26T02:01:49.420254+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:38:22.316897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.533812+0000","last_change":"2024-03-26T02:01:14.949178+0000","last_active":"2024-03-26T02:01:49.533812+0000","last_peered":"2024-03-26T02:01:49.533812+0000","last_clean":"2024-03-26T02:01:49.533812+0000","last_became_active":"2024-03-26T02:01:14.948873+0000","last_became_peered":"2024-03-26T02:01:14.948873+0000","last_unstale":"2024-03-26T02:01:49.533812+0000","last_undegraded":"2024-03-26T02:01:49.533812+0000","last_fullsized":"2024-03-26T02:01:49.533812+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:31:50.292510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"60'2","reported_seq":45,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.428411+0000","last_change":"2024-03-26T02:01:08.215864+0000","last_active":"2024-03-26T02:01:49.428411+0000","last_peered":"2024-03-26T02:01:49.428411+0000","last_clean":"2024-03-26T02:01:49.428411+0000","last_became_active":"2024-03-26T02:01:08.215567+0000","last_became_peered":"2024-03-26T02:01:08.215567+0000","last_unstale":"2024-03-26T02:01:49.428411+0000","last_undegraded":"2024-03-26T02:01:49.428411+0000","last_fullsized":"2024-03-26T02:01:49.428411+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:42:59.899652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"67'29","reported_seq":80,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.533470+0000","last_change":"2024-03-26T02:01:10.220906+0000","last_active":"2024-03-26T02:01:49.533470+0000","last_peered":"2024-03-26T02:01:49.533470+0000","last_clean":"2024-03-26T02:01:49.533470+0000","last_became_active":"2024-03-26T02:01:10.220534+0000","last_became_peered":"2024-03-26T02:01:10.220534+0000","last_unstale":"2024-03-26T02:01:49.533470+0000","last_undegraded":"2024-03-26T02:01:49.533470+0000","last_fullsized":"2024-03-26T02:01:49.533470+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:05:56.519589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:02:23.263547+0000","last_change":"2024-03-26T02:01:12.246468+0000","last_active":"2024-03-26T02:02:23.263547+0000","last_peered":"2024-03-26T02:02:23.263547+0000","last_clean":"2024-03-26T02:02:23.263547+0000","last_became_active":"2024-03-26T02:01:12.246168+0000","last_became_peered":"2024-03-26T02:01:12.246168+0000","last_unstale":"2024-03-26T02:02:23.263547+0000","last_undegraded":"2024-03-26T02:02:23.263547+0000","last_fullsized":"2024-03-26T02:02:23.263547+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:17:48.341193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419388+0000","last_change":"2024-03-26T02:01:14.258131+0000","last_active":"2024-03-26T02:01:49.419388+0000","last_peered":"2024-03-26T02:01:49.419388+0000","last_clean":"2024-03-26T02:01:49.419388+0000","last_became_active":"2024-03-26T02:01:14.257378+0000","last_became_peered":"2024-03-26T02:01:14.257378+0000","last_unstale":"2024-03-26T02:01:49.419388+0000","last_undegraded":"2024-03-26T02:01:49.419388+0000","last_fullsized":"2024-03-26T02:01:49.419388+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:10:51.748424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.428359+0000","last_change":"2024-03-26T02:01:08.217120+0000","last_active":"2024-03-26T02:01:49.428359+0000","last_peered":"2024-03-26T02:01:49.428359+0000","last_clean":"2024-03-26T02:01:49.428359+0000","last_became_active":"2024-03-26T02:01:08.216554+0000","last_became_peered":"2024-03-26T02:01:08.216554+0000","last_unstale":"2024-03-26T02:01:49.428359+0000","last_undegraded":"2024-03-26T02:01:49.428359+0000","last_fullsized":"2024-03-26T02:01:49.428359+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:54:25.381293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"67'24","reported_seq":98,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.420393+0000","last_change":"2024-03-26T02:01:28.912740+0000","last_active":"2024-03-26T02:01:49.420393+0000","last_peered":"2024-03-26T02:01:49.420393+0000","last_clean":"2024-03-26T02:01:49.420393+0000","last_became_active":"2024-03-26T02:01:28.898207+0000","last_became_peered":"2024-03-26T02:01:28.898207+0000","last_unstale":"2024-03-26T02:01:49.420393+0000","last_undegraded":"2024-03-26T02:01:49.420393+0000","last_fullsized":"2024-03-26T02:01:49.420393+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:27:06.140652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.421536+0000","last_change":"2024-03-26T02:01:12.239996+0000","last_active":"2024-03-26T02:01:49.421536+0000","last_peered":"2024-03-26T02:01:49.421536+0000","last_clean":"2024-03-26T02:01:49.421536+0000","last_became_active":"2024-03-26T02:01:12.239686+0000","last_became_peered":"2024-03-26T02:01:12.239686+0000","last_unstale":"2024-03-26T02:01:49.421536+0000","last_undegraded":"2024-03-26T02:01:49.421536+0000","last_fullsized":"2024-03-26T02:01:49.421536+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:50:07.924894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.427565+0000","last_change":"2024-03-26T02:01:14.259206+0000","last_active":"2024-03-26T02:01:49.427565+0000","last_peered":"2024-03-26T02:01:49.427565+0000","last_clean":"2024-03-26T02:01:49.427565+0000","last_became_active":"2024-03-26T02:01:14.258910+0000","last_became_peered":"2024-03-26T02:01:14.258910+0000","last_unstale":"2024-03-26T02:01:49.427565+0000","last_undegraded":"2024-03-26T02:01:49.427565+0000","last_fullsized":"2024-03-26T02:01:49.427565+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:43:01.661996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.421911+0000","last_change":"2024-03-26T02:01:12.242143+0000","last_active":"2024-03-26T02:01:49.421911+0000","last_peered":"2024-03-26T02:01:49.421911+0000","last_clean":"2024-03-26T02:01:49.421911+0000","last_became_active":"2024-03-26T02:01:12.241815+0000","last_became_peered":"2024-03-26T02:01:12.241815+0000","last_unstale":"2024-03-26T02:01:49.421911+0000","last_undegraded":"2024-03-26T02:01:49.421911+0000","last_fullsized":"2024-03-26T02:01:49.421911+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:13:32.222462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"60'1","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.428356+0000","last_change":"2024-03-26T02:01:08.213420+0000","last_active":"2024-03-26T02:01:49.428356+0000","last_peered":"2024-03-26T02:01:49.428356+0000","last_clean":"2024-03-26T02:01:49.428356+0000","last_became_active":"2024-03-26T02:01:08.213157+0000","last_became_peered":"2024-03-26T02:01:08.213157+0000","last_unstale":"2024-03-26T02:01:49.428356+0000","last_undegraded":"2024-03-26T02:01:49.428356+0000","last_fullsized":"2024-03-26T02:01:49.428356+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:02:06.787189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"67'25","reported_seq":71,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.594781+0000","last_change":"2024-03-26T02:01:10.222552+0000","last_active":"2024-03-26T02:01:49.594781+0000","last_peered":"2024-03-26T02:01:49.594781+0000","last_clean":"2024-03-26T02:01:49.594781+0000","last_became_active":"2024-03-26T02:01:10.221462+0000","last_became_peered":"2024-03-26T02:01:10.221462+0000","last_unstale":"2024-03-26T02:01:49.594781+0000","last_undegraded":"2024-03-26T02:01:49.594781+0000","last_fullsized":"2024-03-26T02:01:49.594781+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:51:41.373836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.737745+0000","last_change":"2024-03-26T02:01:14.248293+0000","last_active":"2024-03-26T02:01:49.737745+0000","last_peered":"2024-03-26T02:01:49.737745+0000","last_clean":"2024-03-26T02:01:49.737745+0000","last_became_active":"2024-03-26T02:01:14.247954+0000","last_became_peered":"2024-03-26T02:01:14.247954+0000","last_unstale":"2024-03-26T02:01:49.737745+0000","last_undegraded":"2024-03-26T02:01:49.737745+0000","last_fullsized":"2024-03-26T02:01:49.737745+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:11:35.383448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.421962+0000","last_change":"2024-03-26T02:01:12.235568+0000","last_active":"2024-03-26T02:01:49.421962+0000","last_peered":"2024-03-26T02:01:49.421962+0000","last_clean":"2024-03-26T02:01:49.421962+0000","last_became_active":"2024-03-26T02:01:12.235220+0000","last_became_peered":"2024-03-26T02:01:12.235220+0000","last_unstale":"2024-03-26T02:01:49.421962+0000","last_undegraded":"2024-03-26T02:01:49.421962+0000","last_fullsized":"2024-03-26T02:01:49.421962+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:43:44.371930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.533892+0000","last_change":"2024-03-26T02:01:08.207634+0000","last_active":"2024-03-26T02:01:49.533892+0000","last_peered":"2024-03-26T02:01:49.533892+0000","last_clean":"2024-03-26T02:01:49.533892+0000","last_became_active":"2024-03-26T02:01:08.207339+0000","last_became_peered":"2024-03-26T02:01:08.207339+0000","last_unstale":"2024-03-26T02:01:49.533892+0000","last_undegraded":"2024-03-26T02:01:49.533892+0000","last_fullsized":"2024-03-26T02:01:49.533892+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:18:59.825271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"67'16","reported_seq":58,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739868+0000","last_change":"2024-03-26T02:01:10.221814+0000","last_active":"2024-03-26T02:01:49.739868+0000","last_peered":"2024-03-26T02:01:49.739868+0000","last_clean":"2024-03-26T02:01:49.739868+0000","last_became_active":"2024-03-26T02:01:10.221512+0000","last_became_peered":"2024-03-26T02:01:10.221512+0000","last_unstale":"2024-03-26T02:01:49.739868+0000","last_undegraded":"2024-03-26T02:01:49.739868+0000","last_fullsized":"2024-03-26T02:01:49.739868+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:26:41.564898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419423+0000","last_change":"2024-03-26T02:01:14.254416+0000","last_active":"2024-03-26T02:01:49.419423+0000","last_peered":"2024-03-26T02:01:49.419423+0000","last_clean":"2024-03-26T02:01:49.419423+0000","last_became_active":"2024-03-26T02:01:14.254096+0000","last_became_peered":"2024-03-26T02:01:14.254096+0000","last_unstale":"2024-03-26T02:01:49.419423+0000","last_undegraded":"2024-03-26T02:01:49.419423+0000","last_fullsized":"2024-03-26T02:01:49.419423+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:42:00.911783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":42,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.533603+0000","last_change":"2024-03-26T02:01:12.234741+0000","last_active":"2024-03-26T02:01:49.533603+0000","last_peered":"2024-03-26T02:01:49.533603+0000","last_clean":"2024-03-26T02:01:49.533603+0000","last_became_active":"2024-03-26T02:01:08.212363+0000","last_became_peered":"2024-03-26T02:01:08.212363+0000","last_unstale":"2024-03-26T02:01:49.533603+0000","last_undegraded":"2024-03-26T02:01:49.533603+0000","last_fullsized":"2024-03-26T02:01:49.533603+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:56:33.744663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000429784,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419670+0000","last_change":"2024-03-26T02:01:08.210938+0000","last_active":"2024-03-26T02:01:49.419670+0000","last_peered":"2024-03-26T02:01:49.419670+0000","last_clean":"2024-03-26T02:01:49.419670+0000","last_became_active":"2024-03-26T02:01:08.210662+0000","last_became_peered":"2024-03-26T02:01:08.210662+0000","last_unstale":"2024-03-26T02:01:49.419670+0000","last_undegraded":"2024-03-26T02:01:49.419670+0000","last_fullsized":"2024-03-26T02:01:49.419670+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:11:37.586828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"67'34","reported_seq":106,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:02:23.263374+0000","last_change":"2024-03-26T02:01:10.228402+0000","last_active":"2024-03-26T02:02:23.263374+0000","last_peered":"2024-03-26T02:02:23.263374+0000","last_clean":"2024-03-26T02:02:23.263374+0000","last_became_active":"2024-03-26T02:01:10.227164+0000","last_became_peered":"2024-03-26T02:01:10.227164+0000","last_unstale":"2024-03-26T02:02:23.263374+0000","last_undegraded":"2024-03-26T02:02:23.263374+0000","last_fullsized":"2024-03-26T02:02:23.263374+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:05:11.832807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.738797+0000","last_change":"2024-03-26T02:01:12.245857+0000","last_active":"2024-03-26T02:01:49.738797+0000","last_peered":"2024-03-26T02:01:49.738797+0000","last_clean":"2024-03-26T02:01:49.738797+0000","last_became_active":"2024-03-26T02:01:12.245552+0000","last_became_peered":"2024-03-26T02:01:12.245552+0000","last_unstale":"2024-03-26T02:01:49.738797+0000","last_undegraded":"2024-03-26T02:01:49.738797+0000","last_fullsized":"2024-03-26T02:01:49.738797+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:53:04.495194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.428456+0000","last_change":"2024-03-26T02:01:14.947311+0000","last_active":"2024-03-26T02:01:49.428456+0000","last_peered":"2024-03-26T02:01:49.428456+0000","last_clean":"2024-03-26T02:01:49.428456+0000","last_became_active":"2024-03-26T02:01:14.947010+0000","last_became_peered":"2024-03-26T02:01:14.947010+0000","last_unstale":"2024-03-26T02:01:49.428456+0000","last_undegraded":"2024-03-26T02:01:49.428456+0000","last_fullsized":"2024-03-26T02:01:49.428456+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:12:57.418404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739224+0000","last_change":"2024-03-26T02:01:08.213992+0000","last_active":"2024-03-26T02:01:49.739224+0000","last_peered":"2024-03-26T02:01:49.739224+0000","last_clean":"2024-03-26T02:01:49.739224+0000","last_became_active":"2024-03-26T02:01:08.213716+0000","last_became_peered":"2024-03-26T02:01:08.213716+0000","last_unstale":"2024-03-26T02:01:49.739224+0000","last_undegraded":"2024-03-26T02:01:49.739224+0000","last_fullsized":"2024-03-26T02:01:49.739224+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:37:06.398832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.421692+0000","last_change":"2024-03-26T02:01:12.230250+0000","last_active":"2024-03-26T02:01:49.421692+0000","last_peered":"2024-03-26T02:01:49.421692+0000","last_clean":"2024-03-26T02:01:49.421692+0000","last_became_active":"2024-03-26T02:01:08.205688+0000","last_became_peered":"2024-03-26T02:01:08.205688+0000","last_unstale":"2024-03-26T02:01:49.421692+0000","last_undegraded":"2024-03-26T02:01:49.421692+0000","last_fullsized":"2024-03-26T02:01:49.421692+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:58:48.031942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000498044,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"67'19","reported_seq":68,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.738996+0000","last_change":"2024-03-26T02:01:10.224787+0000","last_active":"2024-03-26T02:01:49.738996+0000","last_peered":"2024-03-26T02:01:49.738996+0000","last_clean":"2024-03-26T02:01:49.738996+0000","last_became_active":"2024-03-26T02:01:10.223876+0000","last_became_peered":"2024-03-26T02:01:10.223876+0000","last_unstale":"2024-03-26T02:01:49.738996+0000","last_undegraded":"2024-03-26T02:01:49.738996+0000","last_fullsized":"2024-03-26T02:01:49.738996+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:23:59.697926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.422065+0000","last_change":"2024-03-26T02:01:12.240414+0000","last_active":"2024-03-26T02:01:49.422065+0000","last_peered":"2024-03-26T02:01:49.422065+0000","last_clean":"2024-03-26T02:01:49.422065+0000","last_became_active":"2024-03-26T02:01:12.240118+0000","last_became_peered":"2024-03-26T02:01:12.240118+0000","last_unstale":"2024-03-26T02:01:49.422065+0000","last_undegraded":"2024-03-26T02:01:49.422065+0000","last_fullsized":"2024-03-26T02:01:49.422065+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:54:59.325332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.534217+0000","last_change":"2024-03-26T02:01:14.253928+0000","last_active":"2024-03-26T02:01:49.534217+0000","last_peered":"2024-03-26T02:01:49.534217+0000","last_clean":"2024-03-26T02:01:49.534217+0000","last_became_active":"2024-03-26T02:01:14.253594+0000","last_became_peered":"2024-03-26T02:01:14.253594+0000","last_unstale":"2024-03-26T02:01:49.534217+0000","last_undegraded":"2024-03-26T02:01:49.534217+0000","last_fullsized":"2024-03-26T02:01:49.534217+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:54:18.338047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"71'128","reported_seq":276,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:51.518377+0000","last_change":"2024-03-26T02:00:57.791957+0000","last_active":"2024-03-26T02:01:51.518377+0000","last_peered":"2024-03-26T02:01:51.518377+0000","last_clean":"2024-03-26T02:01:51.518377+0000","last_became_active":"2024-03-26T02:00:57.791412+0000","last_became_peered":"2024-03-26T02:00:57.791412+0000","last_unstale":"2024-03-26T02:01:51.518377+0000","last_undegraded":"2024-03-26T02:01:51.518377+0000","last_fullsized":"2024-03-26T02:01:51.518377+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T01:58:47.619613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T01:58:47.619613+0000","last_clean_scrub_stamp":"2024-03-26T01:58:47.619613+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:17:46.502230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419222+0000","last_change":"2024-03-26T02:01:08.212292+0000","last_active":"2024-03-26T02:01:49.419222+0000","last_peered":"2024-03-26T02:01:49.419222+0000","last_clean":"2024-03-26T02:01:49.419222+0000","last_became_active":"2024-03-26T02:01:08.211839+0000","last_became_peered":"2024-03-26T02:01:08.211839+0000","last_unstale":"2024-03-26T02:01:49.419222+0000","last_undegraded":"2024-03-26T02:01:49.419222+0000","last_fullsized":"2024-03-26T02:01:49.419222+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:10:02.050298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"67'16","reported_seq":67,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:02:23.263592+0000","last_change":"2024-03-26T02:01:10.223525+0000","last_active":"2024-03-26T02:02:23.263592+0000","last_peered":"2024-03-26T02:02:23.263592+0000","last_clean":"2024-03-26T02:02:23.263592+0000","last_became_active":"2024-03-26T02:01:10.223221+0000","last_became_peered":"2024-03-26T02:01:10.223221+0000","last_unstale":"2024-03-26T02:02:23.263592+0000","last_undegraded":"2024-03-26T02:02:23.263592+0000","last_fullsized":"2024-03-26T02:02:23.263592+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:41:59.197773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.428069+0000","last_change":"2024-03-26T02:01:12.241520+0000","last_active":"2024-03-26T02:01:49.428069+0000","last_peered":"2024-03-26T02:01:49.428069+0000","last_clean":"2024-03-26T02:01:49.428069+0000","last_became_active":"2024-03-26T02:01:12.241240+0000","last_became_peered":"2024-03-26T02:01:12.241240+0000","last_unstale":"2024-03-26T02:01:49.428069+0000","last_undegraded":"2024-03-26T02:01:49.428069+0000","last_fullsized":"2024-03-26T02:01:49.428069+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:22:44.698597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739190+0000","last_change":"2024-03-26T02:01:14.259371+0000","last_active":"2024-03-26T02:01:49.739190+0000","last_peered":"2024-03-26T02:01:49.739190+0000","last_clean":"2024-03-26T02:01:49.739190+0000","last_became_active":"2024-03-26T02:01:14.259028+0000","last_became_peered":"2024-03-26T02:01:14.259028+0000","last_unstale":"2024-03-26T02:01:49.739190+0000","last_undegraded":"2024-03-26T02:01:49.739190+0000","last_fullsized":"2024-03-26T02:01:49.739190+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:24:26.238493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"60'1","reported_seq":39,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739030+0000","last_change":"2024-03-26T02:01:08.211304+0000","last_active":"2024-03-26T02:01:49.739030+0000","last_peered":"2024-03-26T02:01:49.739030+0000","last_clean":"2024-03-26T02:01:49.739030+0000","last_became_active":"2024-03-26T02:01:08.210886+0000","last_became_peered":"2024-03-26T02:01:08.210886+0000","last_unstale":"2024-03-26T02:01:49.739030+0000","last_undegraded":"2024-03-26T02:01:49.739030+0000","last_fullsized":"2024-03-26T02:01:49.739030+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:11:45.385952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"67'5","reported_seq":119,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:02:29.354143+0000","last_change":"2024-03-26T02:01:12.234879+0000","last_active":"2024-03-26T02:02:29.354143+0000","last_peered":"2024-03-26T02:02:29.354143+0000","last_clean":"2024-03-26T02:02:29.354143+0000","last_became_active":"2024-03-26T02:01:08.215480+0000","last_became_peered":"2024-03-26T02:01:08.215480+0000","last_unstale":"2024-03-26T02:02:29.354143+0000","last_undegraded":"2024-03-26T02:02:29.354143+0000","last_fullsized":"2024-03-26T02:02:29.354143+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:29:30.600562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081101999999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":76,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"67'12","reported_seq":47,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739475+0000","last_change":"2024-03-26T02:01:10.222556+0000","last_active":"2024-03-26T02:01:49.739475+0000","last_peered":"2024-03-26T02:01:49.739475+0000","last_clean":"2024-03-26T02:01:49.739475+0000","last_became_active":"2024-03-26T02:01:10.222239+0000","last_became_peered":"2024-03-26T02:01:10.222239+0000","last_unstale":"2024-03-26T02:01:49.739475+0000","last_undegraded":"2024-03-26T02:01:49.739475+0000","last_fullsized":"2024-03-26T02:01:49.739475+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:39:23.747045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.534451+0000","last_change":"2024-03-26T02:01:12.241505+0000","last_active":"2024-03-26T02:01:49.534451+0000","last_peered":"2024-03-26T02:01:49.534451+0000","last_clean":"2024-03-26T02:01:49.534451+0000","last_became_active":"2024-03-26T02:01:12.241236+0000","last_became_peered":"2024-03-26T02:01:12.241236+0000","last_unstale":"2024-03-26T02:01:49.534451+0000","last_undegraded":"2024-03-26T02:01:49.534451+0000","last_fullsized":"2024-03-26T02:01:49.534451+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:05:19.694248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.737938+0000","last_change":"2024-03-26T02:01:14.947343+0000","last_active":"2024-03-26T02:01:49.737938+0000","last_peered":"2024-03-26T02:01:49.737938+0000","last_clean":"2024-03-26T02:01:49.737938+0000","last_became_active":"2024-03-26T02:01:14.946957+0000","last_became_peered":"2024-03-26T02:01:14.946957+0000","last_unstale":"2024-03-26T02:01:49.737938+0000","last_undegraded":"2024-03-26T02:01:49.737938+0000","last_fullsized":"2024-03-26T02:01:49.737938+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:06:19.792424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739435+0000","last_change":"2024-03-26T02:01:08.211850+0000","last_active":"2024-03-26T02:01:49.739435+0000","last_peered":"2024-03-26T02:01:49.739435+0000","last_clean":"2024-03-26T02:01:49.739435+0000","last_became_active":"2024-03-26T02:01:08.211515+0000","last_became_peered":"2024-03-26T02:01:08.211515+0000","last_unstale":"2024-03-26T02:01:49.739435+0000","last_undegraded":"2024-03-26T02:01:49.739435+0000","last_fullsized":"2024-03-26T02:01:49.739435+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:00:03.829850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"67'25","reported_seq":77,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739659+0000","last_change":"2024-03-26T02:01:10.217804+0000","last_active":"2024-03-26T02:01:49.739659+0000","last_peered":"2024-03-26T02:01:49.739659+0000","last_clean":"2024-03-26T02:01:49.739659+0000","last_became_active":"2024-03-26T02:01:10.217469+0000","last_became_peered":"2024-03-26T02:01:10.217469+0000","last_unstale":"2024-03-26T02:01:49.739659+0000","last_undegraded":"2024-03-26T02:01:49.739659+0000","last_fullsized":"2024-03-26T02:01:49.739659+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:10:20.623390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":24,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.594958+0000","last_change":"2024-03-26T02:01:12.241024+0000","last_active":"2024-03-26T02:01:49.594958+0000","last_peered":"2024-03-26T02:01:49.594958+0000","last_clean":"2024-03-26T02:01:49.594958+0000","last_became_active":"2024-03-26T02:01:12.240748+0000","last_became_peered":"2024-03-26T02:01:12.240748+0000","last_unstale":"2024-03-26T02:01:49.594958+0000","last_undegraded":"2024-03-26T02:01:49.594958+0000","last_fullsized":"2024-03-26T02:01:49.594958+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:47:52.744694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.737634+0000","last_change":"2024-03-26T02:01:14.259290+0000","last_active":"2024-03-26T02:01:49.737634+0000","last_peered":"2024-03-26T02:01:49.737634+0000","last_clean":"2024-03-26T02:01:49.737634+0000","last_became_active":"2024-03-26T02:01:14.258916+0000","last_became_peered":"2024-03-26T02:01:14.258916+0000","last_unstale":"2024-03-26T02:01:49.737634+0000","last_undegraded":"2024-03-26T02:01:49.737634+0000","last_fullsized":"2024-03-26T02:01:49.737634+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:19:29.730587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.534124+0000","last_change":"2024-03-26T02:01:08.213557+0000","last_active":"2024-03-26T02:01:49.534124+0000","last_peered":"2024-03-26T02:01:49.534124+0000","last_clean":"2024-03-26T02:01:49.534124+0000","last_became_active":"2024-03-26T02:01:08.213200+0000","last_became_peered":"2024-03-26T02:01:08.213200+0000","last_unstale":"2024-03-26T02:01:49.534124+0000","last_undegraded":"2024-03-26T02:01:49.534124+0000","last_fullsized":"2024-03-26T02:01:49.534124+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:54:26.117881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"67'16","reported_seq":56,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.738505+0000","last_change":"2024-03-26T02:01:10.227638+0000","last_active":"2024-03-26T02:01:49.738505+0000","last_peered":"2024-03-26T02:01:49.738505+0000","last_clean":"2024-03-26T02:01:49.738505+0000","last_became_active":"2024-03-26T02:01:10.227235+0000","last_became_peered":"2024-03-26T02:01:10.227235+0000","last_unstale":"2024-03-26T02:01:49.738505+0000","last_undegraded":"2024-03-26T02:01:49.738505+0000","last_fullsized":"2024-03-26T02:01:49.738505+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:01:21.288308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:02:23.263333+0000","last_change":"2024-03-26T02:01:12.247746+0000","last_active":"2024-03-26T02:02:23.263333+0000","last_peered":"2024-03-26T02:02:23.263333+0000","last_clean":"2024-03-26T02:02:23.263333+0000","last_became_active":"2024-03-26T02:01:12.247388+0000","last_became_peered":"2024-03-26T02:01:12.247388+0000","last_unstale":"2024-03-26T02:02:23.263333+0000","last_undegraded":"2024-03-26T02:02:23.263333+0000","last_fullsized":"2024-03-26T02:02:23.263333+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:20:09.567057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.428086+0000","last_change":"2024-03-26T02:01:14.950129+0000","last_active":"2024-03-26T02:01:49.428086+0000","last_peered":"2024-03-26T02:01:49.428086+0000","last_clean":"2024-03-26T02:01:49.428086+0000","last_became_active":"2024-03-26T02:01:14.949518+0000","last_became_peered":"2024-03-26T02:01:14.949518+0000","last_unstale":"2024-03-26T02:01:49.428086+0000","last_undegraded":"2024-03-26T02:01:49.428086+0000","last_fullsized":"2024-03-26T02:01:49.428086+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:32:44.073092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"60'1","reported_seq":34,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739660+0000","last_change":"2024-03-26T02:01:08.209583+0000","last_active":"2024-03-26T02:01:49.739660+0000","last_peered":"2024-03-26T02:01:49.739660+0000","last_clean":"2024-03-26T02:01:49.739660+0000","last_became_active":"2024-03-26T02:01:08.209301+0000","last_became_peered":"2024-03-26T02:01:08.209301+0000","last_unstale":"2024-03-26T02:01:49.739660+0000","last_undegraded":"2024-03-26T02:01:49.739660+0000","last_fullsized":"2024-03-26T02:01:49.739660+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:16:57.355869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"67'20","reported_seq":62,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419741+0000","last_change":"2024-03-26T02:01:10.221627+0000","last_active":"2024-03-26T02:01:49.419741+0000","last_peered":"2024-03-26T02:01:49.419741+0000","last_clean":"2024-03-26T02:01:49.419741+0000","last_became_active":"2024-03-26T02:01:10.221328+0000","last_became_peered":"2024-03-26T02:01:10.221328+0000","last_unstale":"2024-03-26T02:01:49.419741+0000","last_undegraded":"2024-03-26T02:01:49.419741+0000","last_fullsized":"2024-03-26T02:01:49.419741+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:13:13.086773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:02:23.263303+0000","last_change":"2024-03-26T02:01:12.240289+0000","last_active":"2024-03-26T02:02:23.263303+0000","last_peered":"2024-03-26T02:02:23.263303+0000","last_clean":"2024-03-26T02:02:23.263303+0000","last_became_active":"2024-03-26T02:01:12.239975+0000","last_became_peered":"2024-03-26T02:01:12.239975+0000","last_unstale":"2024-03-26T02:02:23.263303+0000","last_undegraded":"2024-03-26T02:02:23.263303+0000","last_fullsized":"2024-03-26T02:02:23.263303+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T03:15:45.672112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419594+0000","last_change":"2024-03-26T02:01:14.253107+0000","last_active":"2024-03-26T02:01:49.419594+0000","last_peered":"2024-03-26T02:01:49.419594+0000","last_clean":"2024-03-26T02:01:49.419594+0000","last_became_active":"2024-03-26T02:01:14.252756+0000","last_became_peered":"2024-03-26T02:01:14.252756+0000","last_unstale":"2024-03-26T02:01:49.419594+0000","last_undegraded":"2024-03-26T02:01:49.419594+0000","last_fullsized":"2024-03-26T02:01:49.419594+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:46:36.520189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419954+0000","last_change":"2024-03-26T02:01:08.213368+0000","last_active":"2024-03-26T02:01:49.419954+0000","last_peered":"2024-03-26T02:01:49.419954+0000","last_clean":"2024-03-26T02:01:49.419954+0000","last_became_active":"2024-03-26T02:01:08.213072+0000","last_became_peered":"2024-03-26T02:01:08.213072+0000","last_unstale":"2024-03-26T02:01:49.419954+0000","last_undegraded":"2024-03-26T02:01:49.419954+0000","last_fullsized":"2024-03-26T02:01:49.419954+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:37:00.672363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"67'30","reported_seq":85,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.418547+0000","last_change":"2024-03-26T02:01:10.222574+0000","last_active":"2024-03-26T02:01:49.418547+0000","last_peered":"2024-03-26T02:01:49.418547+0000","last_clean":"2024-03-26T02:01:49.418547+0000","last_became_active":"2024-03-26T02:01:10.220256+0000","last_became_peered":"2024-03-26T02:01:10.220256+0000","last_unstale":"2024-03-26T02:01:49.418547+0000","last_undegraded":"2024-03-26T02:01:49.418547+0000","last_fullsized":"2024-03-26T02:01:49.418547+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:03:18.794657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419437+0000","last_change":"2024-03-26T02:01:12.244042+0000","last_active":"2024-03-26T02:01:49.419437+0000","last_peered":"2024-03-26T02:01:49.419437+0000","last_clean":"2024-03-26T02:01:49.419437+0000","last_became_active":"2024-03-26T02:01:12.242628+0000","last_became_peered":"2024-03-26T02:01:12.242628+0000","last_unstale":"2024-03-26T02:01:49.419437+0000","last_undegraded":"2024-03-26T02:01:49.419437+0000","last_fullsized":"2024-03-26T02:01:49.419437+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:21:22.900048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.533913+0000","last_change":"2024-03-26T02:01:14.252827+0000","last_active":"2024-03-26T02:01:49.533913+0000","last_peered":"2024-03-26T02:01:49.533913+0000","last_clean":"2024-03-26T02:01:49.533913+0000","last_became_active":"2024-03-26T02:01:14.252528+0000","last_became_peered":"2024-03-26T02:01:14.252528+0000","last_unstale":"2024-03-26T02:01:49.533913+0000","last_undegraded":"2024-03-26T02:01:49.533913+0000","last_fullsized":"2024-03-26T02:01:49.533913+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:49:38.186864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:02:23.263543+0000","last_change":"2024-03-26T02:01:12.243196+0000","last_active":"2024-03-26T02:02:23.263543+0000","last_peered":"2024-03-26T02:02:23.263543+0000","last_clean":"2024-03-26T02:02:23.263543+0000","last_became_active":"2024-03-26T02:01:12.242907+0000","last_became_peered":"2024-03-26T02:01:12.242907+0000","last_unstale":"2024-03-26T02:02:23.263543+0000","last_undegraded":"2024-03-26T02:02:23.263543+0000","last_fullsized":"2024-03-26T02:02:23.263543+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:48:21.562936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.418523+0000","last_change":"2024-03-26T02:01:08.209782+0000","last_active":"2024-03-26T02:01:49.418523+0000","last_peered":"2024-03-26T02:01:49.418523+0000","last_clean":"2024-03-26T02:01:49.418523+0000","last_became_active":"2024-03-26T02:01:08.209487+0000","last_became_peered":"2024-03-26T02:01:08.209487+0000","last_unstale":"2024-03-26T02:01:49.418523+0000","last_undegraded":"2024-03-26T02:01:49.418523+0000","last_fullsized":"2024-03-26T02:01:49.418523+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:57:12.003049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"67'15","reported_seq":54,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739114+0000","last_change":"2024-03-26T02:01:10.227401+0000","last_active":"2024-03-26T02:01:49.739114+0000","last_peered":"2024-03-26T02:01:49.739114+0000","last_clean":"2024-03-26T02:01:49.739114+0000","last_became_active":"2024-03-26T02:01:10.227113+0000","last_became_peered":"2024-03-26T02:01:10.227113+0000","last_unstale":"2024-03-26T02:01:49.739114+0000","last_undegraded":"2024-03-26T02:01:49.739114+0000","last_fullsized":"2024-03-26T02:01:49.739114+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:57:37.814294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.418798+0000","last_change":"2024-03-26T02:01:14.947358+0000","last_active":"2024-03-26T02:01:49.418798+0000","last_peered":"2024-03-26T02:01:49.418798+0000","last_clean":"2024-03-26T02:01:49.418798+0000","last_became_active":"2024-03-26T02:01:14.946985+0000","last_became_peered":"2024-03-26T02:01:14.946985+0000","last_unstale":"2024-03-26T02:01:49.418798+0000","last_undegraded":"2024-03-26T02:01:49.418798+0000","last_fullsized":"2024-03-26T02:01:49.418798+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:29:11.018800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.534404+0000","last_change":"2024-03-26T02:01:12.247847+0000","last_active":"2024-03-26T02:01:49.534404+0000","last_peered":"2024-03-26T02:01:49.534404+0000","last_clean":"2024-03-26T02:01:49.534404+0000","last_became_active":"2024-03-26T02:01:12.247522+0000","last_became_peered":"2024-03-26T02:01:12.247522+0000","last_unstale":"2024-03-26T02:01:49.534404+0000","last_undegraded":"2024-03-26T02:01:49.534404+0000","last_fullsized":"2024-03-26T02:01:49.534404+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:07:11.520915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419443+0000","last_change":"2024-03-26T02:01:08.209092+0000","last_active":"2024-03-26T02:01:49.419443+0000","last_peered":"2024-03-26T02:01:49.419443+0000","last_clean":"2024-03-26T02:01:49.419443+0000","last_became_active":"2024-03-26T02:01:08.207895+0000","last_became_peered":"2024-03-26T02:01:08.207895+0000","last_unstale":"2024-03-26T02:01:49.419443+0000","last_undegraded":"2024-03-26T02:01:49.419443+0000","last_fullsized":"2024-03-26T02:01:49.419443+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:50:14.560374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"67'17","reported_seq":60,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.420483+0000","last_change":"2024-03-26T02:01:10.226787+0000","last_active":"2024-03-26T02:01:49.420483+0000","last_peered":"2024-03-26T02:01:49.420483+0000","last_clean":"2024-03-26T02:01:49.420483+0000","last_became_active":"2024-03-26T02:01:10.226092+0000","last_became_peered":"2024-03-26T02:01:10.226092+0000","last_unstale":"2024-03-26T02:01:49.420483+0000","last_undegraded":"2024-03-26T02:01:49.420483+0000","last_fullsized":"2024-03-26T02:01:49.420483+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:05:30.553189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.422431+0000","last_change":"2024-03-26T02:01:14.253413+0000","last_active":"2024-03-26T02:01:49.422431+0000","last_peered":"2024-03-26T02:01:49.422431+0000","last_clean":"2024-03-26T02:01:49.422431+0000","last_became_active":"2024-03-26T02:01:14.253115+0000","last_became_peered":"2024-03-26T02:01:14.253115+0000","last_unstale":"2024-03-26T02:01:49.422431+0000","last_undegraded":"2024-03-26T02:01:49.422431+0000","last_fullsized":"2024-03-26T02:01:49.422431+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:31:39.354661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.739519+0000","last_change":"2024-03-26T02:01:12.241559+0000","last_active":"2024-03-26T02:01:49.739519+0000","last_peered":"2024-03-26T02:01:49.739519+0000","last_clean":"2024-03-26T02:01:49.739519+0000","last_became_active":"2024-03-26T02:01:12.241211+0000","last_became_peered":"2024-03-26T02:01:12.241211+0000","last_unstale":"2024-03-26T02:01:49.739519+0000","last_undegraded":"2024-03-26T02:01:49.739519+0000","last_fullsized":"2024-03-26T02:01:49.739519+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:56:54.174990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":32,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.594551+0000","last_change":"2024-03-26T02:01:08.215824+0000","last_active":"2024-03-26T02:01:49.594551+0000","last_peered":"2024-03-26T02:01:49.594551+0000","last_clean":"2024-03-26T02:01:49.594551+0000","last_became_active":"2024-03-26T02:01:08.215539+0000","last_became_peered":"2024-03-26T02:01:08.215539+0000","last_unstale":"2024-03-26T02:01:49.594551+0000","last_undegraded":"2024-03-26T02:01:49.594551+0000","last_fullsized":"2024-03-26T02:01:49.594551+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:17:49.285575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"67'17","reported_seq":81,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.421142+0000","last_change":"2024-03-26T02:01:29.159078+0000","last_active":"2024-03-26T02:01:49.421142+0000","last_peered":"2024-03-26T02:01:49.421142+0000","last_clean":"2024-03-26T02:01:49.421142+0000","last_became_active":"2024-03-26T02:01:29.150851+0000","last_became_peered":"2024-03-26T02:01:29.150851+0000","last_unstale":"2024-03-26T02:01:49.421142+0000","last_undegraded":"2024-03-26T02:01:49.421142+0000","last_fullsized":"2024-03-26T02:01:49.421142+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:07:25.101349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.420640+0000","last_change":"2024-03-26T02:01:14.249417+0000","last_active":"2024-03-26T02:01:49.420640+0000","last_peered":"2024-03-26T02:01:49.420640+0000","last_clean":"2024-03-26T02:01:49.420640+0000","last_became_active":"2024-03-26T02:01:14.249132+0000","last_became_peered":"2024-03-26T02:01:14.249132+0000","last_unstale":"2024-03-26T02:01:49.420640+0000","last_undegraded":"2024-03-26T02:01:49.420640+0000","last_fullsized":"2024-03-26T02:01:49.420640+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:13.227034+0000","last_clean_scrub_stamp":"2024-03-26T02:01:13.227034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:20:11.616069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"67'8","reported_seq":50,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:02:23.263333+0000","last_change":"2024-03-26T02:01:12.243689+0000","last_active":"2024-03-26T02:02:23.263333+0000","last_peered":"2024-03-26T02:02:23.263333+0000","last_clean":"2024-03-26T02:02:23.263333+0000","last_became_active":"2024-03-26T02:01:12.242674+0000","last_became_peered":"2024-03-26T02:01:12.242674+0000","last_unstale":"2024-03-26T02:02:23.263333+0000","last_undegraded":"2024-03-26T02:02:23.263333+0000","last_fullsized":"2024-03-26T02:02:23.263333+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":63,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:11.214405+0000","last_clean_scrub_stamp":"2024-03-26T02:01:11.214405+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:38:39.771989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.419688+0000","last_change":"2024-03-26T02:01:08.209891+0000","last_active":"2024-03-26T02:01:49.419688+0000","last_peered":"2024-03-26T02:01:49.419688+0000","last_clean":"2024-03-26T02:01:49.419688+0000","last_became_active":"2024-03-26T02:01:08.209596+0000","last_became_peered":"2024-03-26T02:01:08.209596+0000","last_unstale":"2024-03-26T02:01:49.419688+0000","last_undegraded":"2024-03-26T02:01:49.419688+0000","last_fullsized":"2024-03-26T02:01:49.419688+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:07.185295+0000","last_clean_scrub_stamp":"2024-03-26T02:01:07.185295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:20:03.580395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"67'10","reported_seq":44,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-03-26T02:01:49.420126+0000","last_change":"2024-03-26T02:01:10.226753+0000","last_active":"2024-03-26T02:01:49.420126+0000","last_peered":"2024-03-26T02:01:49.420126+0000","last_clean":"2024-03-26T02:01:49.420126+0000","last_became_active":"2024-03-26T02:01:10.224630+0000","last_became_peered":"2024-03-26T02:01:10.224630+0000","last_unstale":"2024-03-26T02:01:49.420126+0000","last_undegraded":"2024-03-26T02:01:49.420126+0000","last_fullsized":"2024-03-26T02:01:49.420126+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T02:01:09.198642+0000","last_clean_scrub_stamp":"2024-03-26T02:01:09.198642+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T02:06:50.698862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":590,"ondisk_log_size":590,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":71,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234005,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28292,"kb_used_data":1272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716860,"statfs":{"total":95995035648,"available":95966064640,"internally_reserved":0,"allocated":1302528,"data_stored":868050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495641,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28272,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716880,"statfs":{"total":95995035648,"available":95966085120,"internally_reserved":0,"allocated":1286144,"data_stored":867011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31900,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713252,"statfs":{"total":95995035648,"available":95962370048,"internally_reserved":0,"allocated":737280,"data_stored":277687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32460,"kb_used_data":1280,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712692,"statfs":{"total":95995035648,"available":95961796608,"internally_reserved":0,"allocated":1310720,"data_stored":868282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31852,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713300,"statfs":{"total":95995035648,"available":95962419200,"internally_reserved":0,"allocated":688128,"data_stored":276567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182442,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31892,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713260,"statfs":{"total":95995035648,"available":95962378240,"internally_reserved":0,"allocated":729088,"data_stored":278011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018917,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27720,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":716800,"data_stored":277712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757279,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27700,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717452,"statfs":{"total":95995035648,"available":95966670848,"internally_reserved":0,"allocated":696320,"data_stored":277066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T02:02:34.464 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-26T02:02:34.464 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-26T02:02:34.464 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-26T02:02:34.464 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-26T02:02:34.708 INFO:teuthology.orchestra.run.smithi072.stderr:Inferring config /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T02:02:35.104 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:02:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:02:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:02:35.413 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:35 smithi072 ceph-mon[30177]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:35.413 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:35 smithi072 ceph-mon[30177]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:02:35.414 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:35 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:02:35.414 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:35 smithi072 ceph-mon[41619]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:35.414 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:35 smithi072 ceph-mon[41619]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:02:35.414 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:35 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:02:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:35 smithi173 ceph-mon[33839]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:35 smithi173 ceph-mon[33839]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T02:02:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:35 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:02:37.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:37 smithi072 ceph-mon[30177]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:37.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:37 smithi072 ceph-mon[41619]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:37 smithi173 ceph-mon[33839]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:38.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.242+0000 7f897e89e700 1 -- 172.21.15.72:0/2070892682 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8978100670 msgr2=0x7f8978100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:38.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.242+0000 7f897e89e700 1 --2- 172.21.15.72:0/2070892682 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8978100670 0x7f8978100af0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f896c009990 tx=0x7f896c009ca0 comp rx=0 tx=0).stop 2024-03-26T02:02:38.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.242+0000 7f897e89e700 1 -- 172.21.15.72:0/2070892682 shutdown_connections 2024-03-26T02:02:38.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.242+0000 7f897e89e700 1 --2- 172.21.15.72:0/2070892682 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8978101030 0x7f897810ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:38.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.242+0000 7f897e89e700 1 --2- 172.21.15.72:0/2070892682 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f89781070f0 0x7f89781074d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:38.245 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.242+0000 7f897e89e700 1 --2- 172.21.15.72:0/2070892682 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8978100670 0x7f8978100af0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:38.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.242+0000 7f897e89e700 1 -- 172.21.15.72:0/2070892682 >> 172.21.15.72:0/2070892682 conn(0x7f8978077580 msgr2=0x7f89780fe9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:38.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.242+0000 7f897e89e700 1 -- 172.21.15.72:0/2070892682 shutdown_connections 2024-03-26T02:02:38.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.242+0000 7f897e89e700 1 -- 172.21.15.72:0/2070892682 wait complete. 2024-03-26T02:02:38.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.243+0000 7f897e89e700 1 Processor -- start 2024-03-26T02:02:38.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.243+0000 7f897e89e700 1 -- start start 2024-03-26T02:02:38.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.243+0000 7f897e89e700 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8978100670 0x7f8978197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:38.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.243+0000 7f897e89e700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8978101030 0x7f8978198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:38.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.243+0000 7f897e89e700 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f89781070f0 0x7f897819c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:38.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.243+0000 7f897e89e700 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897819cce0 con 0x7f8978101030 2024-03-26T02:02:38.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.243+0000 7f897e89e700 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897819ce50 con 0x7f8978100670 2024-03-26T02:02:38.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.243+0000 7f897e89e700 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f897819d010 con 0x7f89781070f0 2024-03-26T02:02:38.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.243+0000 7f89777fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8978101030 0x7f8978198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:38.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.243+0000 7f89777fe700 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8978101030 0x7f8978198390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:33194/0 (socket says 172.21.15.72:33194) 2024-03-26T02:02:38.246 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.243+0000 7f89777fe700 1 -- 172.21.15.72:0/375185356 learned_addr learned my addr 172.21.15.72:0/375185356 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:02:38.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.243+0000 7f897ce3b700 1 --2- 172.21.15.72:0/375185356 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f89781070f0 0x7f897819c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:38.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.243+0000 7f8977fff700 1 --2- 172.21.15.72:0/375185356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8978100670 0x7f8978197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:38.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.243+0000 7f89777fe700 1 -- 172.21.15.72:0/375185356 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f89781070f0 msgr2=0x7f897819c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:38.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.243+0000 7f89777fe700 1 --2- 172.21.15.72:0/375185356 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f89781070f0 0x7f897819c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:38.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.243+0000 7f89777fe700 1 -- 172.21.15.72:0/375185356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8978100670 msgr2=0x7f8978197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:38.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.243+0000 7f89777fe700 1 --2- 172.21.15.72:0/375185356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8978100670 0x7f8978197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:38.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.243+0000 7f89777fe700 1 -- 172.21.15.72:0/375185356 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f896c0095f0 con 0x7f8978101030 2024-03-26T02:02:38.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.243+0000 7f897ce3b700 1 --2- 172.21.15.72:0/375185356 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f89781070f0 0x7f897819c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:02:38.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.244+0000 7f89777fe700 1 --2- 172.21.15.72:0/375185356 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8978101030 0x7f8978198390 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f896c005910 tx=0x7f896c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:38.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.244+0000 7f89757fa700 1 -- 172.21.15.72:0/375185356 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f896c01d070 con 0x7f8978101030 2024-03-26T02:02:38.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.244+0000 7f89757fa700 1 -- 172.21.15.72:0/375185356 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f896c00b610 con 0x7f8978101030 2024-03-26T02:02:38.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.244+0000 7f897e89e700 1 -- 172.21.15.72:0/375185356 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f897819d290 con 0x7f8978101030 2024-03-26T02:02:38.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.245+0000 7f89757fa700 1 -- 172.21.15.72:0/375185356 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f896c0173f0 con 0x7f8978101030 2024-03-26T02:02:38.247 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.245+0000 7f897e89e700 1 -- 172.21.15.72:0/375185356 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8978101950 con 0x7f8978101030 2024-03-26T02:02:38.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.245+0000 7f89757fa700 1 -- 172.21.15.72:0/375185356 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7f896c017550 con 0x7f8978101030 2024-03-26T02:02:38.248 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.245+0000 7f897e89e700 1 -- 172.21.15.72:0/375185356 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8978068d40 con 0x7f8978101030 2024-03-26T02:02:38.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.246+0000 7f89757fa700 1 --2- 172.21.15.72:0/375185356 >> v2:172.21.15.72:6800/3246690770 conn(0x7f896006da80 0x7f896006ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:02:38.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.246+0000 7f89757fa700 1 -- 172.21.15.72:0/375185356 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6259+0+0 (secure 0 0 0) 0x7f896c08e5d0 con 0x7f8978101030 2024-03-26T02:02:38.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.246+0000 7f8977fff700 1 --2- 172.21.15.72:0/375185356 >> v2:172.21.15.72:6800/3246690770 conn(0x7f896006da80 0x7f896006ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:02:38.249 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.246+0000 7f8977fff700 1 --2- 172.21.15.72:0/375185356 >> v2:172.21.15.72:6800/3246690770 conn(0x7f896006da80 0x7f896006ff40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8968010d80 tx=0x7f8968012040 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:02:38.251 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.251+0000 7f89757fa700 1 -- 172.21.15.72:0/375185356 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f896c05bb20 con 0x7f8978101030 2024-03-26T02:02:38.505 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.504+0000 7f897e89e700 1 -- 172.21.15.72:0/375185356 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8978199160 con 0x7f8978101030 2024-03-26T02:02:38.506 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.505+0000 7f89757fa700 1 -- 172.21.15.72:0/375185356 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f896c05b6b0 con 0x7f8978101030 2024-03-26T02:02:38.506 INFO:teuthology.orchestra.run.smithi072.stdout: 2024-03-26T02:02:38.506 INFO:teuthology.orchestra.run.smithi072.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-26T02:02:38.507 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.507+0000 7f897e89e700 1 -- 172.21.15.72:0/375185356 >> v2:172.21.15.72:6800/3246690770 conn(0x7f896006da80 msgr2=0x7f896006ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:38.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.507+0000 7f897e89e700 1 --2- 172.21.15.72:0/375185356 >> v2:172.21.15.72:6800/3246690770 conn(0x7f896006da80 0x7f896006ff40 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8968010d80 tx=0x7f8968012040 comp rx=0 tx=0).stop 2024-03-26T02:02:38.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.507+0000 7f897e89e700 1 -- 172.21.15.72:0/375185356 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8978101030 msgr2=0x7f8978198390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:02:38.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.507+0000 7f897e89e700 1 --2- 172.21.15.72:0/375185356 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8978101030 0x7f8978198390 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f896c005910 tx=0x7f896c004730 comp rx=0 tx=0).stop 2024-03-26T02:02:38.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.507+0000 7f897e89e700 1 -- 172.21.15.72:0/375185356 shutdown_connections 2024-03-26T02:02:38.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.507+0000 7f897e89e700 1 --2- 172.21.15.72:0/375185356 >> v2:172.21.15.72:6800/3246690770 conn(0x7f896006da80 0x7f896006ff40 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:38.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.507+0000 7f897e89e700 1 --2- 172.21.15.72:0/375185356 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f89781070f0 0x7f897819c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:38.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.507+0000 7f897e89e700 1 --2- 172.21.15.72:0/375185356 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8978100670 0x7f8978197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:38.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.507+0000 7f897e89e700 1 --2- 172.21.15.72:0/375185356 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8978101030 0x7f8978198390 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:02:38.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.507+0000 7f897e89e700 1 -- 172.21.15.72:0/375185356 >> 172.21.15.72:0/375185356 conn(0x7f8978077580 msgr2=0x7f8978104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:02:38.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.507+0000 7f897e89e700 1 -- 172.21.15.72:0/375185356 shutdown_connections 2024-03-26T02:02:38.508 INFO:teuthology.orchestra.run.smithi072.stderr:2024-03-26T02:02:38.507+0000 7f897e89e700 1 -- 172.21.15.72:0/375185356 wait complete. 2024-03-26T02:02:39.273 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:39 smithi072 ceph-mon[30177]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:39.273 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/375185356' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T02:02:39.273 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:39 smithi072 ceph-mon[41619]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:39.273 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/375185356' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T02:02:39.407 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-26T02:02:39.408 INFO:tasks.cephadm:Setup complete, yielding 2024-03-26T02:02:39.408 INFO:teuthology.run_tasks:Running task workunit... 2024-03-26T02:02:39.422 INFO:tasks.workunit:Pulling workunits from ref 9f04c91b77e4619963d41730d8cf756e3a6152de 2024-03-26T02:02:39.423 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-26T02:02:39.423 DEBUG:teuthology.orchestra.run.smithi072:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-26T02:02:39.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:02:39.439 INFO:teuthology.orchestra.run.smithi072.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-26T02:02:39.439 DEBUG:teuthology.orchestra.run.smithi072:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-26T02:02:39.494 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-26T02:02:39.494 DEBUG:teuthology.orchestra.run.smithi072:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-26T02:02:39.549 INFO:tasks.workunit:timeout=3h 2024-03-26T02:02:39.549 INFO:tasks.workunit:cleanup=True 2024-03-26T02:02:39.550 DEBUG:teuthology.orchestra.run.smithi072:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 9f04c91b77e4619963d41730d8cf756e3a6152de 2024-03-26T02:02:39.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:39 smithi173 ceph-mon[33839]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:39.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/375185356' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T02:02:39.604 INFO:tasks.workunit.client.0.smithi072.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-26T02:02:40.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:02:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:02:40.969 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:02:40 smithi072 systemd[1]: Starting Ceph alertmanager.a for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:02:41.395 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:41 smithi072 ceph-mon[30177]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:41.396 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:41 smithi072 ceph-mon[41619]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:41.396 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:02:41 smithi072 podman[80242]: 2024-03-26 02:02:41.014630839 +0000 UTC m=+0.017836914 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-26T02:02:41.396 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:02:41 smithi072 podman[80242]: 2024-03-26 02:02:41.117128617 +0000 UTC m=+0.120334676 volume create 8cca6ae3d1e68fad4df02b37a88c5af564e18bc723445c20786f4f55fc7722cd 2024-03-26T02:02:41.396 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:02:41 smithi072 podman[80242]: 2024-03-26 02:02:41.209964477 +0000 UTC m=+0.213170536 container create a6a4516addd902082335d581ea869269c4ccfb60a2154c5877fe7f3e9ebad00d (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:02:41.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:41 smithi173 ceph-mon[33839]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:41.829 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:02:41 smithi072 podman[80242]: 2024-03-26 02:02:41.499501042 +0000 UTC m=+0.502707100 container init a6a4516addd902082335d581ea869269c4ccfb60a2154c5877fe7f3e9ebad00d (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:02:41.829 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:02:41 smithi072 podman[80242]: 2024-03-26 02:02:41.505162831 +0000 UTC m=+0.508368890 container start a6a4516addd902082335d581ea869269c4ccfb60a2154c5877fe7f3e9ebad00d (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:02:41.829 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:02:41 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a[80257]: ts=2024-03-26T02:02:41.516Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-26T02:02:41.829 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:02:41 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a[80257]: ts=2024-03-26T02:02:41.516Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-26T02:02:41.829 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:02:41 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a[80257]: ts=2024-03-26T02:02:41.517Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.72 port=9094 2024-03-26T02:02:41.830 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:02:41 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a[80257]: ts=2024-03-26T02:02:41.518Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T02:02:41.830 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:02:41 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a[80257]: ts=2024-03-26T02:02:41.542Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T02:02:41.830 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:02:41 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a[80257]: ts=2024-03-26T02:02:41.542Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T02:02:41.830 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:02:41 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a[80257]: ts=2024-03-26T02:02:41.546Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-26T02:02:41.830 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:02:41 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a[80257]: ts=2024-03-26T02:02:41.546Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-26T02:02:41.830 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:02:41 smithi072 bash[80242]: a6a4516addd902082335d581ea869269c4ccfb60a2154c5877fe7f3e9ebad00d 2024-03-26T02:02:41.830 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:02:41 smithi072 systemd[1]: Started Ceph alertmanager.a for c86fbdd4-eb13-11ee-95d0-87774f69a715. 2024-03-26T02:02:42.118 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:02:41 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:02:41] ENGINE Bus STOPPING 2024-03-26T02:02:42.118 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:02:41 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:02:41] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T02:02:42.118 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:02:41 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:02:41] ENGINE Bus STOPPED 2024-03-26T02:02:42.118 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:02:41 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:02:41] ENGINE Bus STARTING 2024-03-26T02:02:42.118 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:02:41 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:02:41] ENGINE Serving on http://:::9283 2024-03-26T02:02:42.119 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:02:41 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:02:41] ENGINE Bus STARTED 2024-03-26T02:02:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:42 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:02:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:42 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:42 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:42 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:42 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:42 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:42 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:42.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:42 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T02:02:42.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:42 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:42.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:42 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:02:42.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:42 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:42.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:42 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:42.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:42 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:42.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:42 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:42.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:42 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:42.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:42 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:42.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:42 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T02:02:42.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:42 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:42 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:02:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:42 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:42 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:42 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:42 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:42 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:42 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:42 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T02:02:42.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:42 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:43 smithi072 ceph-mon[30177]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:43 smithi072 ceph-mon[30177]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T02:02:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:43 smithi072 ceph-mon[30177]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T02:02:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:43 smithi072 ceph-mon[30177]: Deploying daemon grafana.a on smithi173 2024-03-26T02:02:43.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:43 smithi072 ceph-mon[41619]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:43.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:43 smithi072 ceph-mon[41619]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T02:02:43.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:43 smithi072 ceph-mon[41619]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T02:02:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:43 smithi072 ceph-mon[41619]: Deploying daemon grafana.a on smithi173 2024-03-26T02:02:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:43 smithi173 ceph-mon[33839]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:43 smithi173 ceph-mon[33839]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T02:02:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:43 smithi173 ceph-mon[33839]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T02:02:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:43 smithi173 ceph-mon[33839]: Deploying daemon grafana.a on smithi173 2024-03-26T02:02:43.969 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:02:43 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a[80257]: ts=2024-03-26T02:02:43.519Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000974235s 2024-03-26T02:02:45.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:45 smithi072 ceph-mon[30177]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:45.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:45 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:45.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:02:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:02:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:02:45.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:45 smithi072 ceph-mon[41619]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:45.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:45 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:45.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:45 smithi173 ceph-mon[33839]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:45 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:47.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:47 smithi072 ceph-mon[30177]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:47.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:47 smithi072 ceph-mon[41619]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:47.562 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:47 smithi173 ceph-mon[33839]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:49 smithi072 ceph-mon[30177]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:49.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:49 smithi072 ceph-mon[41619]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:49.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:49 smithi173 ceph-mon[33839]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:50.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:02:50.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:02:50.494 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:02:50.812 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:02:50 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:02:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:51 smithi072 ceph-mon[30177]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:51.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:51 smithi072 ceph-mon[41619]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:51.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:51 smithi173 ceph-mon[33839]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:51.969 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:02:51 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a[80257]: ts=2024-03-26T02:02:51.522Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003449439s 2024-03-26T02:02:52.430 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:52 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:02:52.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:52 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:02:52.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:52 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:02:53.423 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:53 smithi173 ceph-mon[33839]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:53 smithi072 ceph-mon[30177]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:53.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:53 smithi072 ceph-mon[41619]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:54.813 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:54 smithi173 systemd[1]: Starting Ceph grafana.a for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:02:55.100 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:54 smithi173 podman[66757]: 2024-03-26 02:02:54.744539542 +0000 UTC m=+0.017076603 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-26T02:02:55.100 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:54 smithi173 podman[66757]: 2024-03-26 02:02:54.855239255 +0000 UTC m=+0.127776289 container create 46d8593c9f17ea9e200b1c07b724f77b9a660540bc02c29658c4cf2ef80d3dbe (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a, version=8.5, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , name=ubi8, io.buildah.version=1.32.0, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-04-29T11:48:11.055897, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, release=239.1651231664, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, distribution-scope=public, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container) 2024-03-26T02:02:55.100 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 podman[66757]: 2024-03-26 02:02:55.09970158 +0000 UTC m=+0.372238618 container init 46d8593c9f17ea9e200b1c07b724f77b9a660540bc02c29658c4cf2ef80d3dbe (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a, release=239.1651231664, description=Ceph Grafana Container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, vcs-type=git, io.openshift.tags=base rhel8, version=8.5, io.openshift.expose-services=, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, distribution-scope=public, maintainer=Paul Cuzner , name=ubi8, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0) 2024-03-26T02:02:55.351 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-mon[33839]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:55.351 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 podman[66757]: 2024-03-26 02:02:55.104940821 +0000 UTC m=+0.377477863 container start 46d8593c9f17ea9e200b1c07b724f77b9a660540bc02c29658c4cf2ef80d3dbe (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , io.buildah.version=1.32.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, release=239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, vendor=Red Hat, Inc., distribution-scope=public, name=ubi8, description=Ceph Grafana Container, io.openshift.tags=base rhel8, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-04-29T11:48:11.055897) 2024-03-26T02:02:55.351 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 bash[66757]: 46d8593c9f17ea9e200b1c07b724f77b9a660540bc02c29658c4cf2ef80d3dbe 2024-03-26T02:02:55.351 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 systemd[1]: Started Ceph grafana.a for c86fbdd4-eb13-11ee-95d0-87774f69a715. 2024-03-26T02:02:55.352 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=settings t=2024-03-26T02:02:55.264343205Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-26T02:02:55.352 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=settings t=2024-03-26T02:02:55.264521013Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-26T02:02:55.352 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=settings t=2024-03-26T02:02:55.264537569Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-26T02:02:55.352 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=settings t=2024-03-26T02:02:55.264544039Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-26T02:02:55.352 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=settings t=2024-03-26T02:02:55.26454984Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-26T02:02:55.352 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=settings t=2024-03-26T02:02:55.26455551Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-26T02:02:55.352 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=settings t=2024-03-26T02:02:55.264560732Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-26T02:02:55.352 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=settings t=2024-03-26T02:02:55.264566044Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-26T02:02:55.352 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=settings t=2024-03-26T02:02:55.26457138Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-26T02:02:55.352 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=settings t=2024-03-26T02:02:55.26457728Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-26T02:02:55.354 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=settings t=2024-03-26T02:02:55.26458229Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-26T02:02:55.354 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=settings t=2024-03-26T02:02:55.26458715Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-26T02:02:55.354 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=settings t=2024-03-26T02:02:55.26459225Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-26T02:02:55.354 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=settings t=2024-03-26T02:02:55.264597493Z level=info msg="App mode production" 2024-03-26T02:02:55.354 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=sqlstore t=2024-03-26T02:02:55.264638931Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-26T02:02:55.355 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=sqlstore t=2024-03-26T02:02:55.264653763Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-26T02:02:55.355 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.265543898Z level=info msg="Starting DB migrations" 2024-03-26T02:02:55.355 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.265622446Z level=info msg="Executing migration" id="create migration_log table" 2024-03-26T02:02:55.355 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.266154108Z level=info msg="Executing migration" id="create user table" 2024-03-26T02:02:55.355 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.26662105Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-26T02:02:55.355 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.267114966Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-26T02:02:55.355 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.267564806Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-26T02:02:55.355 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.268103998Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-26T02:02:55.355 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.26852087Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-26T02:02:55.355 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.269266002Z level=info msg="Executing migration" id="create user table v2" 2024-03-26T02:02:55.356 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.269677453Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-26T02:02:55.356 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.270074872Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-26T02:02:55.356 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.270524989Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-26T02:02:55.356 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.270850672Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-26T02:02:55.356 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.27122576Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-26T02:02:55.356 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.271640058Z level=info msg="Executing migration" id="Update user table charset" 2024-03-26T02:02:55.356 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.271917306Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-26T02:02:55.356 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.272376516Z level=info msg="Executing migration" id="Add missing user data" 2024-03-26T02:02:55.357 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.272739952Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-26T02:02:55.357 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.273172486Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-26T02:02:55.357 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.27357858Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-26T02:02:55.357 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.27397234Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-26T02:02:55.357 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.275098517Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-26T02:02:55.357 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.275544972Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-26T02:02:55.357 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.276073998Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-26T02:02:55.357 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.276511498Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-26T02:02:55.357 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.276976267Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-26T02:02:55.357 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.277529635Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-26T02:02:55.357 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.277880067Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-26T02:02:55.357 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.27837874Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-26T02:02:55.358 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.278861299Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-26T02:02:55.358 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.279344947Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-26T02:02:55.358 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.279849506Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-26T02:02:55.358 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.280565448Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-26T02:02:55.359 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.280957936Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-26T02:02:55.359 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.281357099Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-26T02:02:55.359 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.28175592Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-26T02:02:55.359 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.282312236Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-26T02:02:55.359 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.28284843Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-26T02:02:55.359 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.283339485Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-26T02:02:55.359 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.283786045Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-26T02:02:55.359 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.284165529Z level=info msg="Executing migration" id="create star table" 2024-03-26T02:02:55.360 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.284526062Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-26T02:02:55.360 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.284920575Z level=info msg="Executing migration" id="create org table v1" 2024-03-26T02:02:55.360 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.285337624Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-26T02:02:55.360 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.287827395Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-26T02:02:55.360 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.288256151Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-26T02:02:55.360 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.288644308Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-26T02:02:55.360 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.289035787Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-26T02:02:55.360 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.289410357Z level=info msg="Executing migration" id="Update org table charset" 2024-03-26T02:02:55.361 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.289660183Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-26T02:02:55.361 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.289913439Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-26T02:02:55.361 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.290220649Z level=info msg="Executing migration" id="create dashboard table" 2024-03-26T02:02:55.361 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.290589043Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-26T02:02:55.361 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.290956403Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-26T02:02:55.361 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.291350902Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-26T02:02:55.362 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.291756172Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-26T02:02:55.362 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.294870855Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-26T02:02:55.362 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.295333901Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-26T02:02:55.362 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.298404358Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-26T02:02:55.362 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.301210443Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-26T02:02:55.362 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.30405609Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-26T02:02:55.362 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.304499672Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-26T02:02:55.362 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.304833019Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-26T02:02:55.362 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.307572938Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-26T02:02:55.362 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.310083695Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-26T02:02:55.362 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.310593673Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-26T02:02:55.362 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.311098684Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-26T02:02:55.362 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.311604006Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-26T02:02:55.363 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.31404295Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-26T02:02:55.363 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.314570451Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-26T02:02:55.363 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.316608839Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-26T02:02:55.363 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.31716023Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-26T02:02:55.363 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.31746789Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-26T02:02:55.363 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.317769212Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-26T02:02:55.363 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.319504432Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-26T02:02:55.363 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.32207854Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-26T02:02:55.364 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.324669546Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-26T02:02:55.364 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.327336755Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-26T02:02:55.364 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.327690953Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-26T02:02:55.364 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.328470819Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-26T02:02:55.364 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.329507998Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-26T02:02:55.364 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.330165492Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-26T02:02:55.364 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.330617458Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-26T02:02:55.364 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.331503916Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-26T02:02:55.364 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.332653491Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-26T02:02:55.364 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.333432475Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-26T02:02:55.364 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.333894613Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-26T02:02:55.364 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.334800594Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-26T02:02:55.365 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.335502743Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-26T02:02:55.365 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.336418583Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-26T02:02:55.365 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.337114535Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-26T02:02:55.366 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.338043619Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-26T02:02:55.366 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.33874413Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-26T02:02:55.366 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.33948592Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-26T02:02:55.366 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.339950597Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-26T02:02:55.366 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.342844795Z level=info msg="Executing migration" id="create data_source table" 2024-03-26T02:02:55.366 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.345515889Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-26T02:02:55.366 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.348240682Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-26T02:02:55.366 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.351147083Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-26T02:02:55.366 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.351583777Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-26T02:02:55.366 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.354230907Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-26T02:02:55.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:55 smithi072 ceph-mon[30177]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:55.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:55 smithi072 ceph-mon[41619]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:55.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:02:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:02:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:02:55.603 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.355247875Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-26T02:02:55.603 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.356972851Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-26T02:02:55.603 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.359706455Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-26T02:02:55.604 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.360147465Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-26T02:02:55.604 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.362833972Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-26T02:02:55.604 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.365748808Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-26T02:02:55.604 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.368361026Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-26T02:02:55.604 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.368686458Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-26T02:02:55.604 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.37135429Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-26T02:02:55.604 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.372108188Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-26T02:02:55.604 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.374042327Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-26T02:02:55.604 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.376556223Z level=info msg="Executing migration" id="Add uid column" 2024-03-26T02:02:55.604 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.377101202Z level=info msg="Executing migration" id="Update uid value" 2024-03-26T02:02:55.605 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.379776277Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-26T02:02:55.605 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.382560922Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-26T02:02:55.605 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.384923414Z level=info msg="Executing migration" id="create api_key table" 2024-03-26T02:02:55.605 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.385366765Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-26T02:02:55.605 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.388036764Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-26T02:02:55.605 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.390474971Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-26T02:02:55.605 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.393027619Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-26T02:02:55.605 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.393462313Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-26T02:02:55.605 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.395888503Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-26T02:02:55.605 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.396323725Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-26T02:02:55.605 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.398697047Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-26T02:02:55.606 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.401207004Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-26T02:02:55.606 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.403708082Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-26T02:02:55.606 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.406259432Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-26T02:02:55.606 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.408868972Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-26T02:02:55.606 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.409618038Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-26T02:02:55.606 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.410364501Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-26T02:02:55.606 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.411320925Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-26T02:02:55.606 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.411956255Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-26T02:02:55.606 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.412737623Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-26T02:02:55.606 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.413475009Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-26T02:02:55.606 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.414050224Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-26T02:02:55.606 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.414652884Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-26T02:02:55.607 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.415361361Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-26T02:02:55.607 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.415765315Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-26T02:02:55.607 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.416172961Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-26T02:02:55.608 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.416997445Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-26T02:02:55.608 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.417907384Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-26T02:02:55.608 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.418585454Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-26T02:02:55.608 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.419400763Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-26T02:02:55.608 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.419872254Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-26T02:02:55.608 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.420493597Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-26T02:02:55.608 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.421317199Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-26T02:02:55.608 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.422139901Z level=info msg="Executing migration" id="create quota table v1" 2024-03-26T02:02:55.608 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.422924299Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-26T02:02:55.609 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.42382707Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-26T02:02:55.609 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.424552866Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-26T02:02:55.609 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.425509878Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-26T02:02:55.609 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.426164461Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-26T02:02:55.609 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.426969852Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-26T02:02:55.609 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.427788414Z level=info msg="Executing migration" id="create session table" 2024-03-26T02:02:55.609 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.42871099Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-26T02:02:55.609 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.429348294Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-26T02:02:55.609 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.430176907Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-26T02:02:55.609 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.431143636Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-26T02:02:55.609 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.431809432Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-26T02:02:55.609 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.432672427Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-26T02:02:55.609 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.433194901Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-26T02:02:55.609 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.433568462Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-26T02:02:55.610 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.434445607Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-26T02:02:55.610 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.434992668Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-26T02:02:55.610 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.437217466Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-26T02:02:55.610 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.439902689Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-26T02:02:55.610 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.442735624Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-26T02:02:55.610 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.445611923Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-26T02:02:55.610 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.448303523Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-26T02:02:55.610 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.450905001Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-26T02:02:55.610 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.453771928Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-26T02:02:55.610 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.456387034Z level=info msg="Executing migration" id="create alert table v1" 2024-03-26T02:02:55.610 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.459335905Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-26T02:02:55.610 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.462187525Z level=info msg="Executing migration" id="add index alert state" 2024-03-26T02:02:55.610 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.464782619Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-26T02:02:55.611 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.4675267Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-26T02:02:55.611 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.470308755Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-26T02:02:55.611 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.472760666Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-26T02:02:55.611 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.475632993Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-26T02:02:55.611 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.478112711Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-26T02:02:55.611 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.480665013Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-26T02:02:55.611 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.483222259Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-26T02:02:55.611 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.485800279Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-26T02:02:55.611 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.488400805Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-26T02:02:55.611 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.491240512Z level=info msg="Executing migration" id="Add column is_default" 2024-03-26T02:02:55.611 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.493805749Z level=info msg="Executing migration" id="Add column frequency" 2024-03-26T02:02:55.611 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.496212401Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-26T02:02:55.611 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.49881305Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-26T02:02:55.611 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.501378691Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-26T02:02:55.612 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.50390888Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-26T02:02:55.612 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.50628773Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-26T02:02:55.612 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.509089803Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-26T02:02:55.612 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.511767962Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-26T02:02:55.612 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.514457095Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-26T02:02:55.612 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.516912833Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-26T02:02:55.612 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.519332216Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-26T02:02:55.612 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.521983926Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-26T02:02:55.612 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.524686747Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-26T02:02:55.612 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.525521076Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-26T02:02:55.612 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.526406958Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-26T02:02:55.612 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.527123284Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-26T02:02:55.612 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.527905716Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-26T02:02:55.612 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.528666719Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-26T02:02:55.612 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.52940721Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-26T02:02:55.613 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.530327589Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-26T02:02:55.613 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.531009944Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-26T02:02:55.613 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.531748756Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-26T02:02:55.613 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.532530647Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-26T02:02:55.613 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.533297286Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-26T02:02:55.613 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.534098913Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-26T02:02:55.613 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.535091956Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-26T02:02:55.613 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.535708544Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-26T02:02:55.613 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.536515487Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-26T02:02:55.613 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.537315441Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-26T02:02:55.613 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.537980518Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-26T02:02:55.613 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.538757825Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-26T02:02:55.613 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.53964826Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-26T02:02:55.613 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.54050245Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-26T02:02:55.613 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.541418756Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-26T02:02:55.614 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.542163188Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-26T02:02:55.614 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.543792531Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-26T02:02:55.614 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.54475924Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-26T02:02:55.614 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.545550393Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-26T02:02:55.614 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.546503616Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-26T02:02:55.614 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.547204358Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-26T02:02:55.614 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.548111796Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-26T02:02:55.614 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.548936585Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-26T02:02:55.614 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.549796735Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-26T02:02:55.614 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.550584485Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-26T02:02:55.614 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.551451656Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-26T02:02:55.614 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.552259588Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-26T02:02:55.614 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.553153956Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-26T02:02:55.614 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.554052511Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-26T02:02:55.615 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.554998555Z level=info msg="Executing migration" id="Move region to single row" 2024-03-26T02:02:55.615 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.55584917Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-26T02:02:55.615 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.556623293Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-26T02:02:55.615 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.557538219Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-26T02:02:55.615 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.55829519Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-26T02:02:55.615 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.561170877Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-26T02:02:55.615 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.563946493Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-26T02:02:55.615 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.566818143Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-26T02:02:55.615 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.569395263Z level=info msg="Executing migration" id="create test_data table" 2024-03-26T02:02:55.615 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.572125861Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-26T02:02:55.615 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.572524158Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-26T02:02:55.615 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.575472566Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-26T02:02:55.615 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.578503855Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-26T02:02:55.615 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.581106374Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-26T02:02:55.615 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.583817821Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-26T02:02:55.616 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.586796071Z level=info msg="Executing migration" id="create team table" 2024-03-26T02:02:55.616 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.58953386Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-26T02:02:55.616 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.592422555Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-26T02:02:55.616 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.595035187Z level=info msg="Executing migration" id="create team member table" 2024-03-26T02:02:55.616 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.597948557Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-26T02:02:55.616 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.600563053Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-26T02:02:55.616 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.603306878Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-26T02:02:55.616 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.606065617Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-26T02:02:55.853 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.608903214Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-26T02:02:55.854 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.611595914Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-26T02:02:55.854 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.614271553Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-26T02:02:55.854 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.617135743Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-26T02:02:55.854 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.619977391Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-26T02:02:55.854 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.620420342Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-26T02:02:55.854 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.622613718Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-26T02:02:55.854 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.625234883Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-26T02:02:55.854 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.628096778Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-26T02:02:55.854 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.628583886Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-26T02:02:55.854 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.631415056Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-26T02:02:55.854 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.634146318Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-26T02:02:55.854 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.636862684Z level=info msg="Executing migration" id="create tag table" 2024-03-26T02:02:55.854 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.637365838Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-26T02:02:55.855 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.640230677Z level=info msg="Executing migration" id="create login attempt table" 2024-03-26T02:02:55.855 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.642904782Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-26T02:02:55.855 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.645665283Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-26T02:02:55.855 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.646151478Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-26T02:02:55.855 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.649085525Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-26T02:02:55.855 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.649773866Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-26T02:02:55.855 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.650586142Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-26T02:02:55.855 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.650931843Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-26T02:02:55.855 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.651715263Z level=info msg="Executing migration" id="create user auth table" 2024-03-26T02:02:55.855 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.652586526Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-26T02:02:55.855 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.653433211Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-26T02:02:55.855 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.653761169Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-26T02:02:55.855 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.654615187Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-26T02:02:55.855 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.655429435Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-26T02:02:55.855 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.656385124Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-26T02:02:55.856 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.657364129Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-26T02:02:55.856 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.658320948Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-26T02:02:55.856 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.659195903Z level=info msg="Executing migration" id="create server_lock table" 2024-03-26T02:02:55.856 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.660080603Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-26T02:02:55.856 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.660579895Z level=info msg="Executing migration" id="create user auth token table" 2024-03-26T02:02:55.856 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.661299111Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-26T02:02:55.856 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.662258588Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-26T02:02:55.856 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.662961395Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-26T02:02:55.856 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.663413835Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-26T02:02:55.858 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.664294303Z level=info msg="Executing migration" id="create cache_data table" 2024-03-26T02:02:55.858 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.665140034Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-26T02:02:55.858 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.666123214Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-26T02:02:55.858 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.666701683Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-26T02:02:55.858 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.667505045Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-26T02:02:55.858 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.668437298Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-26T02:02:55.858 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.668863037Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-26T02:02:55.858 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.669327051Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-26T02:02:55.858 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.670059044Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-26T02:02:55.858 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.670507748Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-26T02:02:55.859 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.67079325Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-26T02:02:55.859 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.671240687Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-26T02:02:55.859 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.671676422Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-26T02:02:55.862 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.672092225Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-26T02:02:55.862 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.672533739Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-26T02:02:55.862 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.673343843Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-26T02:02:55.862 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.673794282Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-26T02:02:55.862 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.674088693Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-26T02:02:55.862 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.674514192Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-26T02:02:55.862 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.674932959Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-26T02:02:55.862 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.675374721Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-26T02:02:55.862 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.675684267Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-26T02:02:55.862 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.67611659Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-26T02:02:55.862 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.676557906Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-26T02:02:55.862 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.676968578Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-26T02:02:55.863 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.677434109Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-26T02:02:55.863 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.678294024Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-26T02:02:55.863 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.678722152Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-26T02:02:55.863 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.679158006Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-26T02:02:55.863 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.68259906Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-26T02:02:55.863 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.685847706Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-26T02:02:55.863 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.686349915Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-26T02:02:55.863 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.686785552Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-26T02:02:55.863 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.687643794Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-26T02:02:55.863 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.688110067Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-26T02:02:55.863 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.688539363Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-26T02:02:55.863 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.68894918Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-26T02:02:55.863 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.689388557Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-26T02:02:55.864 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.689672675Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-26T02:02:55.864 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.690507832Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-26T02:02:55.864 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.691336711Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-26T02:02:55.864 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.692122304Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-26T02:02:55.864 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.692530811Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-26T02:02:55.864 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.692923705Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-26T02:02:55.864 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.693757659Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-26T02:02:55.864 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.694609057Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-26T02:02:55.864 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.695040132Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-26T02:02:55.864 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.695869484Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-26T02:02:55.864 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.696723556Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-26T02:02:55.864 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.697035161Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-26T02:02:55.864 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.697501669Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-26T02:02:55.864 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.698277304Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-26T02:02:55.865 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.698757056Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-26T02:02:55.865 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.69902074Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-26T02:02:55.865 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.699897346Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-26T02:02:55.865 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.700756449Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-26T02:02:55.865 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.701634519Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-26T02:02:55.865 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.702519892Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-26T02:02:55.865 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.703370153Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-26T02:02:55.865 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.703662342Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-26T02:02:55.865 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.704484863Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-26T02:02:55.865 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.705357192Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-26T02:02:55.865 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.706235146Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-26T02:02:55.865 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.707139569Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-26T02:02:55.865 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.707916272Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-26T02:02:55.865 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.708787894Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-26T02:02:55.866 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.709608218Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-26T02:02:55.866 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.710057638Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-26T02:02:55.866 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.710949153Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-26T02:02:55.866 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.711713983Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-26T02:02:55.866 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.712613959Z level=info msg="Executing migration" id="create alert_image table" 2024-03-26T02:02:55.866 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.713186983Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-26T02:02:55.866 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.713993584Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-26T02:02:55.866 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.714809497Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-26T02:02:55.866 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.7156234Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-26T02:02:55.866 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.71605593Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T02:02:55.867 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.716131197Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T02:02:55.867 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.716818183Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-26T02:02:55.867 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.717763966Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-26T02:02:55.867 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.71856969Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-26T02:02:55.867 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.718830584Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-26T02:02:55.868 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.718942946Z level=info msg="alerts found to migrate" alerts=0 2024-03-26T02:02:55.868 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.719431882Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-26T02:02:55.868 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.722089552Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-26T02:02:55.868 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.724786617Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-26T02:02:55.868 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.727532269Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-26T02:02:55.868 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.728131168Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-26T02:02:55.868 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.730825499Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-26T02:02:55.868 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.733412022Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-26T02:02:55.868 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.736168674Z level=info msg="Executing migration" id="create data_keys table" 2024-03-26T02:02:55.868 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.736678897Z level=info msg="Executing migration" id="create secrets table" 2024-03-26T02:02:55.868 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.739341647Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-26T02:02:55.868 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.743611849Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-26T02:02:55.868 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.74641436Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-26T02:02:55.868 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.746805767Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-26T02:02:55.869 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.75091683Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-26T02:02:55.869 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.755035638Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-26T02:02:55.869 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.757768161Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-26T02:02:55.869 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.758238613Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-26T02:02:55.869 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.761061288Z level=info msg="Executing migration" id="create permission table" 2024-03-26T02:02:55.869 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.763622319Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-26T02:02:55.869 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.766240074Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-26T02:02:55.869 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.766779154Z level=info msg="Executing migration" id="create role table" 2024-03-26T02:02:55.869 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.769643079Z level=info msg="Executing migration" id="add column display_name" 2024-03-26T02:02:55.869 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.771999447Z level=info msg="Executing migration" id="add column group_name" 2024-03-26T02:02:55.869 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.774396292Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-26T02:02:55.869 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.774907046Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-26T02:02:55.869 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.777436395Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-26T02:02:55.869 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.779838214Z level=info msg="Executing migration" id="create team role table" 2024-03-26T02:02:55.870 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.782331372Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-26T02:02:55.870 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.782787572Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-26T02:02:55.870 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.785567964Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-26T02:02:55.870 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.788109468Z level=info msg="Executing migration" id="create user role table" 2024-03-26T02:02:55.870 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.790466479Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-26T02:02:55.870 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.790912676Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-26T02:02:55.870 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.793743553Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-26T02:02:55.870 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.796418309Z level=info msg="Executing migration" id="create builtin role table" 2024-03-26T02:02:55.870 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.79748818Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-26T02:02:55.870 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.79794156Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-26T02:02:55.870 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.798851483Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-26T02:02:55.871 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.80005556Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-26T02:02:55.871 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.800852143Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-26T02:02:55.871 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.801320499Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-26T02:02:55.871 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.802111792Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-26T02:02:55.871 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.80282407Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-26T02:02:55.871 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.803537376Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-26T02:02:55.871 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.804050451Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-26T02:02:55.871 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.80499899Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-26T02:02:55.871 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.805946394Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-26T02:02:55.871 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.806549443Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-26T02:02:55.871 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.806808605Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-26T02:02:55.871 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.807220783Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-26T02:02:55.871 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.807678037Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-26T02:02:55.871 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.808446835Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-26T02:02:55.871 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.808758518Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-26T02:02:55.872 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.809266141Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-26T02:02:55.872 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.809674813Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-26T02:02:55.874 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.81009459Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-26T02:02:55.874 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.811059138Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-26T02:02:55.874 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.811952685Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-26T02:02:55.874 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.812639049Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-26T02:02:55.874 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.813093901Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-26T02:02:55.874 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.81408404Z level=info msg="Executing migration" id="add correlation config column" 2024-03-26T02:02:55.874 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.815155808Z level=info msg="Executing migration" id="create entity_events table" 2024-03-26T02:02:55.874 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.815967841Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-26T02:02:55.874 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.816379341Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T02:02:55.874 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.816466014Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T02:02:55.875 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.817194317Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T02:02:55.875 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.817297277Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T02:02:55.875 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.818108712Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-26T02:02:55.875 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.818719924Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-26T02:02:55.875 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.819142089Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-26T02:02:55.875 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.819965881Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T02:02:55.875 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.820927163Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-26T02:02:55.875 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.823610155Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T02:02:55.875 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.824104889Z level=info msg="Executing migration" id="Drop public config table" 2024-03-26T02:02:55.875 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.826819106Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-26T02:02:55.875 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.829643174Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-26T02:02:55.875 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.832159953Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T02:02:55.875 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.832614115Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-26T02:02:55.876 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.835334085Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-26T02:02:55.876 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.838381731Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-26T02:02:55.876 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.8410518Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-26T02:02:55.876 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.842061917Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-26T02:02:55.876 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.844749098Z level=info msg="Executing migration" id="add share column" 2024-03-26T02:02:55.876 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.847470614Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-26T02:02:55.876 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.850177655Z level=info msg="Executing migration" id="create file table" 2024-03-26T02:02:55.876 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.850663815Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-26T02:02:55.876 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.853528833Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-26T02:02:55.876 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.856183357Z level=info msg="Executing migration" id="create file_meta table" 2024-03-26T02:02:56.230 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.858786325Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-26T02:02:56.231 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.859259717Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-26T02:02:56.231 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.86179599Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-26T02:02:56.231 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.864314523Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-26T02:02:56.231 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.866869883Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-26T02:02:56.231 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.867408286Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-26T02:02:56.231 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.87003256Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-26T02:02:56.231 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.872467158Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-26T02:02:56.231 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.875024782Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-26T02:02:56.231 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.875433709Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-26T02:02:56.231 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.877888614Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-26T02:02:56.231 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.878299127Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-26T02:02:56.231 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.880605281Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-26T02:02:56.231 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.883230026Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-26T02:02:56.232 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.883803471Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-26T02:02:56.232 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.893627282Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-26T02:02:56.232 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.896130852Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-26T02:02:56.232 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.89893198Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-26T02:02:56.232 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.902658888Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-26T02:02:56.232 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.903255083Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-26T02:02:56.232 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.90402808Z level=info msg="Executing migration" id="create folder table" 2024-03-26T02:02:56.232 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.904893585Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-26T02:02:56.232 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.905550933Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-26T02:02:56.233 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.908395775Z level=info msg="Executing migration" id="Update folder title length" 2024-03-26T02:02:56.233 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.911245582Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-26T02:02:56.233 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=migrator t=2024-03-26T02:02:55.913801089Z level=info msg="migrations completed" performed=485 skipped=0 duration=648.203559ms 2024-03-26T02:02:56.233 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=sqlstore t=2024-03-26T02:02:55.914213816Z level=info msg="Created default organization" 2024-03-26T02:02:56.233 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=plugin.loader t=2024-03-26T02:02:55.970506844Z level=info msg="Plugin registered" pluginID=input 2024-03-26T02:02:56.233 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=plugin.loader t=2024-03-26T02:02:55.997076739Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-26T02:02:56.233 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=plugin.loader t=2024-03-26T02:02:55.997099095Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-26T02:02:56.233 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=secrets t=2024-03-26T02:02:55.997142991Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-26T02:02:56.233 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=query_data t=2024-03-26T02:02:55.999221897Z level=info msg="Query Service initialization" 2024-03-26T02:02:56.233 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:56 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=live.push_http t=2024-03-26T02:02:56.001699905Z level=info msg="Live Push Gateway initialization" 2024-03-26T02:02:56.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:56 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:56.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:56 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:56.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:56 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:56.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:56 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:56.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:56 smithi072 ceph-mon[41619]: Deploying daemon node-exporter.a on smithi072 2024-03-26T02:02:56.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:56 smithi072 ceph-mon[41619]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:56.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:56 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:56.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:56 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:56.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:56 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:56.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:56 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:56.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:56 smithi072 ceph-mon[30177]: Deploying daemon node-exporter.a on smithi072 2024-03-26T02:02:56.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:56 smithi072 ceph-mon[30177]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:56.473 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:56 smithi072 bash[80541]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-26T02:02:56.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:56 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:56.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:56 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:56.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:56 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:56.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:56 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:02:56.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:56 smithi173 ceph-mon[33839]: Deploying daemon node-exporter.a on smithi072 2024-03-26T02:02:56.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:56 smithi173 ceph-mon[33839]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:02:57.063 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:56 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=infra.usagestats.collector t=2024-03-26T02:02:56.609735069Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-26T02:02:57.063 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:56 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=provisioning.datasources t=2024-03-26T02:02:56.610728087Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-26T02:02:57.063 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:56 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=provisioning.datasources t=2024-03-26T02:02:56.621756041Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-26T02:02:57.063 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:56 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=provisioning.alerting t=2024-03-26T02:02:56.632544045Z level=info msg="starting to provision alerting" 2024-03-26T02:02:57.064 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:56 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=provisioning.alerting t=2024-03-26T02:02:56.632562394Z level=info msg="finished to provision alerting" 2024-03-26T02:02:57.064 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:56 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=grafanaStorageLogger t=2024-03-26T02:02:56.632717557Z level=info msg="storage starting" 2024-03-26T02:02:57.064 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:56 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=ngalert.state.manager t=2024-03-26T02:02:56.632807767Z level=info msg="Warming state cache for startup" 2024-03-26T02:02:57.064 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:56 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=http.server t=2024-03-26T02:02:56.634606379Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-26T02:02:57.064 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:56 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=ngalert.state.manager t=2024-03-26T02:02:56.638611136Z level=info msg="State cache has been initialized" states=0 duration=5.802681ms 2024-03-26T02:02:57.064 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:56 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=ticker t=2024-03-26T02:02:56.638659374Z level=info msg=starting first_tick=2024-03-26T02:03:00Z 2024-03-26T02:02:57.064 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:56 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=ngalert.multiorg.alertmanager t=2024-03-26T02:02:56.638671765Z level=info msg="starting MultiOrg Alertmanager" 2024-03-26T02:02:57.064 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:02:56 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=sqlstore.transactions t=2024-03-26T02:02:56.654376251Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T02:02:57.219 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:56 smithi072 bash[80541]: Getting image source signatures 2024-03-26T02:02:57.219 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:56 smithi072 bash[80541]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-26T02:02:57.219 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:56 smithi072 bash[80541]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-26T02:02:57.219 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:56 smithi072 bash[80541]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-26T02:02:58.469 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 bash[80541]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T02:02:58.469 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 bash[80541]: Writing manifest to image destination 2024-03-26T02:02:58.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:02:58 smithi173 ceph-mon[33839]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:58.842 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-mon[30177]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:58.842 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 podman[80541]: 2024-03-26 02:02:58.52051516 +0000 UTC m=+2.202005337 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T02:02:58.842 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 podman[80541]: 2024-03-26 02:02:58.612032593 +0000 UTC m=+2.293522770 container create 2568a301a2cb4316d62b6ad51b7630db732204fc359224994f39c41961fdef52 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:02:58.843 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-mon[41619]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:02:59.220 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 podman[80541]: 2024-03-26 02:02:58.8422624 +0000 UTC m=+2.523752577 container init 2568a301a2cb4316d62b6ad51b7630db732204fc359224994f39c41961fdef52 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:02:59.220 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 podman[80541]: 2024-03-26 02:02:58.848079616 +0000 UTC m=+2.529569798 container start 2568a301a2cb4316d62b6ad51b7630db732204fc359224994f39c41961fdef52 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:02:59.220 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.851Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T02:02:59.220 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.851Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T02:02:59.220 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.852Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T02:02:59.220 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.852Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T02:02:59.220 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.852Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T02:02:59.220 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.852Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T02:02:59.220 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T02:02:59.221 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T02:02:59.221 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T02:02:59.221 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T02:02:59.221 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T02:02:59.221 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T02:02:59.221 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T02:02:59.221 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T02:02:59.221 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T02:02:59.221 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T02:02:59.221 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T02:02:59.221 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T02:02:59.221 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T02:02:59.221 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T02:02:59.222 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T02:02:59.222 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T02:02:59.222 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T02:02:59.222 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T02:02:59.222 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T02:02:59.222 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T02:02:59.222 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T02:02:59.222 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T02:02:59.222 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T02:02:59.222 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T02:02:59.222 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T02:02:59.222 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T02:02:59.222 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T02:02:59.222 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T02:02:59.223 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T02:02:59.223 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T02:02:59.223 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T02:02:59.223 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T02:02:59.223 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T02:02:59.223 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T02:02:59.223 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T02:02:59.223 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T02:02:59.223 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T02:02:59.223 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T02:02:59.223 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T02:02:59.223 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T02:02:59.224 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T02:02:59.224 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T02:02:59.224 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T02:02:59.224 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T02:02:59.224 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T02:02:59.224 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T02:02:59.224 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a[80591]: ts=2024-03-26T02:02:58.853Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T02:02:59.224 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 bash[80541]: 2568a301a2cb4316d62b6ad51b7630db732204fc359224994f39c41961fdef52 2024-03-26T02:02:59.224 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:02:58 smithi072 systemd[1]: Started Ceph node-exporter.a for c86fbdd4-eb13-11ee-95d0-87774f69a715. 2024-03-26T02:03:00.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:00 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:00.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:00 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:00.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:00 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:00.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:00 smithi173 ceph-mon[33839]: Deploying daemon node-exporter.b on smithi173 2024-03-26T02:03:00.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:00 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:00.314 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:00 smithi173 bash[67044]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-26T02:03:00.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:00 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:00.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:00 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:00.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:00 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:00.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:00 smithi072 ceph-mon[41619]: Deploying daemon node-exporter.b on smithi173 2024-03-26T02:03:00.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:00 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:00.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:00 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:00.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:00 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:00.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:00 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:00.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:00 smithi072 ceph-mon[30177]: Deploying daemon node-exporter.b on smithi173 2024-03-26T02:03:00.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:00 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:00.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:03:00 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:03:00.813 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:00 smithi173 bash[67044]: Getting image source signatures 2024-03-26T02:03:00.813 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:00 smithi173 bash[67044]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-26T02:03:00.813 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:00 smithi173 bash[67044]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-26T02:03:00.813 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:00 smithi173 bash[67044]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-26T02:03:01.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:01 smithi173 ceph-mon[33839]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:01.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:01 smithi072 ceph-mon[30177]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:01.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:01 smithi072 ceph-mon[41619]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:02.394 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:02 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:03:02.395 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:02 smithi173 bash[67044]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T02:03:02.395 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:02 smithi173 bash[67044]: Writing manifest to image destination 2024-03-26T02:03:02.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:02 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:03:02.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:02 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:03:03.035 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:02 smithi173 podman[67044]: 2024-03-26 02:03:02.692555254 +0000 UTC m=+2.574881323 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T02:03:03.035 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:02 smithi173 podman[67044]: 2024-03-26 02:03:02.774663684 +0000 UTC m=+2.656989795 container create 107338ad9b8535493bb7b1dbc80d15b2fb55ebd87d8f6e068ce93b03cac47345 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:03:03.035 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 podman[67044]: 2024-03-26 02:03:03.035209755 +0000 UTC m=+2.917535825 container init 107338ad9b8535493bb7b1dbc80d15b2fb55ebd87d8f6e068ce93b03cac47345 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:03:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-mon[33839]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:03.313 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 podman[67044]: 2024-03-26 02:03:03.041270258 +0000 UTC m=+2.923596329 container start 107338ad9b8535493bb7b1dbc80d15b2fb55ebd87d8f6e068ce93b03cac47345 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:03:03.313 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.045Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T02:03:03.313 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.045Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T02:03:03.313 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.045Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T02:03:03.314 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.045Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T02:03:03.314 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T02:03:03.314 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T02:03:03.314 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T02:03:03.314 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T02:03:03.314 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T02:03:03.314 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T02:03:03.314 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T02:03:03.314 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T02:03:03.314 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T02:03:03.314 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T02:03:03.314 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T02:03:03.314 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T02:03:03.315 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T02:03:03.315 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T02:03:03.315 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T02:03:03.315 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T02:03:03.315 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T02:03:03.315 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T02:03:03.315 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T02:03:03.315 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T02:03:03.316 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T02:03:03.316 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T02:03:03.316 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T02:03:03.316 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T02:03:03.316 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T02:03:03.316 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T02:03:03.316 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T02:03:03.316 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T02:03:03.316 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T02:03:03.316 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T02:03:03.316 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T02:03:03.316 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T02:03:03.316 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T02:03:03.316 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T02:03:03.316 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T02:03:03.317 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T02:03:03.317 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T02:03:03.317 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T02:03:03.317 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T02:03:03.317 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T02:03:03.317 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T02:03:03.317 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T02:03:03.317 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T02:03:03.317 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T02:03:03.317 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T02:03:03.317 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T02:03:03.317 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.046Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T02:03:03.317 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.047Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T02:03:03.317 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b[67094]: ts=2024-03-26T02:03:03.047Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T02:03:03.317 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 bash[67044]: 107338ad9b8535493bb7b1dbc80d15b2fb55ebd87d8f6e068ce93b03cac47345 2024-03-26T02:03:03.318 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:03:03 smithi173 systemd[1]: Started Ceph node-exporter.b for c86fbdd4-eb13-11ee-95d0-87774f69a715. 2024-03-26T02:03:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:03 smithi072 ceph-mon[30177]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:03.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:03 smithi072 ceph-mon[41619]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:04.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:04 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:04.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:04 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:04.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:04 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:04.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:04 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:04.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:04 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:04.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:04 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:03:04.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:04 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:04.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:04 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:04.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:04 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:04.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:04 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:04.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:04 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:04.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:04 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:03:04.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:04 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:04.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:04 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:04.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:04 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:04.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:04 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:04.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:04 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:04.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:04 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:03:05.297 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:03:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:03:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:03:05.297 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:05 smithi072 ceph-mon[30177]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:05.297 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:05 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:05.297 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:05 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:03:05.298 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:05 smithi072 ceph-mon[41619]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:05.298 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:05 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:05.298 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:05 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:03:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:05 smithi173 ceph-mon[33839]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:05 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:05 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:03:07.422 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:07 smithi173 ceph-mon[33839]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:07.422 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:07 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:07.423 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:07 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:07.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:07 smithi072 ceph-mon[30177]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:07.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:07 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:07.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:07 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:07.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:07 smithi072 ceph-mon[41619]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:07.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:07 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:07.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:07 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:09.172 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:08 smithi072 ceph-mon[30177]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:09.172 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:09.172 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:09.172 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:03:09.172 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:03:09.172 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:09.173 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:08 smithi072 ceph-mon[41619]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:09.173 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:09.173 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:09.173 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:03:09.173 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:03:09.173 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:08 smithi173 ceph-mon[33839]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:03:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:03:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:09.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:09 smithi072 ceph-mon[30177]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T02:03:09.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:09 smithi072 ceph-mon[30177]: Reconfiguring daemon alertmanager.a on smithi072 2024-03-26T02:03:09.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:09 smithi072 ceph-mon[41619]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T02:03:09.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:09 smithi072 ceph-mon[41619]: Reconfiguring daemon alertmanager.a on smithi072 2024-03-26T02:03:10.218 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:09 smithi173 ceph-mon[33839]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T02:03:10.218 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:09 smithi173 ceph-mon[33839]: Reconfiguring daemon alertmanager.a on smithi072 2024-03-26T02:03:10.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:03:10 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:03:10.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:10 smithi072 ceph-mon[30177]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:10.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:10 smithi072 ceph-mon[41619]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:11.238 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:10 smithi173 ceph-mon[33839]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:12.151 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:11 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:03:12.152 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:12 smithi072 systemd[1]: Stopping Ceph alertmanager.a for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:03:12.152 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:11 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:03:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:11 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:03:12.719 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:12 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a[80257]: ts=2024-03-26T02:03:12.430Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-26T02:03:12.719 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:12 smithi072 podman[81424]: 2024-03-26 02:03:12.451060315 +0000 UTC m=+0.138351552 container died a6a4516addd902082335d581ea869269c4ccfb60a2154c5877fe7f3e9ebad00d (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:03:13.109 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:12 smithi072 ceph-mon[30177]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:13.109 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:12 smithi072 ceph-mon[41619]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:13.109 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:12 smithi072 podman[81424]: 2024-03-26 02:03:12.856376111 +0000 UTC m=+0.543667345 container cleanup a6a4516addd902082335d581ea869269c4ccfb60a2154c5877fe7f3e9ebad00d (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:03:13.109 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:12 smithi072 bash[81424]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T02:03:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:12 smithi173 ceph-mon[33839]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:13.469 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:13 smithi072 podman[81441]: 2024-03-26 02:03:13.109303224 +0000 UTC m=+0.661478522 container remove a6a4516addd902082335d581ea869269c4ccfb60a2154c5877fe7f3e9ebad00d (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:03:13.469 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:13 smithi072 podman[81441]: 2024-03-26 02:03:13.202356384 +0000 UTC m=+0.754531684 volume remove 8cca6ae3d1e68fad4df02b37a88c5af564e18bc723445c20786f4f55fc7722cd 2024-03-26T02:03:13.469 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:13 smithi072 systemd[1]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T02:03:13.469 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:13 smithi072 systemd[1]: Stopped Ceph alertmanager.a for c86fbdd4-eb13-11ee-95d0-87774f69a715. 2024-03-26T02:03:13.469 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:13 smithi072 systemd[1]: Starting Ceph alertmanager.a for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:03:13.969 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:13 smithi072 podman[81538]: 2024-03-26 02:03:13.480508098 +0000 UTC m=+0.020888283 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-26T02:03:13.969 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:13 smithi072 podman[81538]: 2024-03-26 02:03:13.581245463 +0000 UTC m=+0.121625620 volume create 306385f69bcab269c1dddabcf459af04dde6b15d3226e44db036681fe7163cf6 2024-03-26T02:03:13.969 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:13 smithi072 podman[81538]: 2024-03-26 02:03:13.69045919 +0000 UTC m=+0.230839347 container create 5ac26ae8cdb85eab56cfd04168026ac56098d12a240f90d46667f8df0a7f3c7c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:03:15.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:14 smithi072 ceph-mon[30177]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:15.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:14 smithi072 ceph-mon[41619]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:15.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:03:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:03:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:03:15.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:14 smithi173 ceph-mon[33839]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:16 smithi072 ceph-mon[30177]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:17.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:16 smithi072 ceph-mon[41619]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:17.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:16 smithi173 ceph-mon[33839]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:18.873 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:18 smithi072 podman[81538]: 2024-03-26 02:03:18.619042407 +0000 UTC m=+5.159422581 container init 5ac26ae8cdb85eab56cfd04168026ac56098d12a240f90d46667f8df0a7f3c7c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:03:18.873 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:18 smithi072 podman[81538]: 2024-03-26 02:03:18.625050378 +0000 UTC m=+5.165430536 container start 5ac26ae8cdb85eab56cfd04168026ac56098d12a240f90d46667f8df0a7f3c7c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:03:18.873 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:18 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a[81552]: ts=2024-03-26T02:03:18.636Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-26T02:03:18.873 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:18 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a[81552]: ts=2024-03-26T02:03:18.637Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-26T02:03:18.874 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:18 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a[81552]: ts=2024-03-26T02:03:18.638Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.72 port=9094 2024-03-26T02:03:18.874 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:18 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a[81552]: ts=2024-03-26T02:03:18.639Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T02:03:18.874 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:18 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a[81552]: ts=2024-03-26T02:03:18.672Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T02:03:18.874 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:18 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a[81552]: ts=2024-03-26T02:03:18.673Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T02:03:18.874 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:18 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a[81552]: ts=2024-03-26T02:03:18.676Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-26T02:03:18.875 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:18 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a[81552]: ts=2024-03-26T02:03:18.676Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-26T02:03:19.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:18 smithi072 ceph-mon[30177]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:19.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:18 smithi072 ceph-mon[41619]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:19.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:18 smithi173 ceph-mon[33839]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:20.146 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:19 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:03:20.147 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:19 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:03:20.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:19 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:03:20.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:03:20 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:03:20.870 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:20 smithi072 bash[81538]: 5ac26ae8cdb85eab56cfd04168026ac56098d12a240f90d46667f8df0a7f3c7c 2024-03-26T02:03:20.870 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:20 smithi072 systemd[1]: Started Ceph alertmanager.a for c86fbdd4-eb13-11ee-95d0-87774f69a715. 2024-03-26T02:03:20.871 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:20 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a[81552]: ts=2024-03-26T02:03:20.640Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000929941s 2024-03-26T02:03:21.218 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:20 smithi072 ceph-mon[30177]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:21.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:20 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:21.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:20 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:21.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:20 smithi072 ceph-mon[41619]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:21.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:20 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:21.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:20 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:21.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:20 smithi173 ceph-mon[33839]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:21.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:20 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:21.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:20 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:22.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:21 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:03:22.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:21 smithi173 ceph-mon[33839]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T02:03:22.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:21 smithi173 ceph-mon[33839]: Reconfiguring daemon prometheus.a on smithi173 2024-03-26T02:03:22.218 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:21 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:03:22.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:21 smithi072 ceph-mon[30177]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T02:03:22.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:21 smithi072 ceph-mon[30177]: Reconfiguring daemon prometheus.a on smithi173 2024-03-26T02:03:22.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:21 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:03:22.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:21 smithi072 ceph-mon[41619]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T02:03:22.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:21 smithi072 ceph-mon[41619]: Reconfiguring daemon prometheus.a on smithi173 2024-03-26T02:03:23.190 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:22 smithi173 ceph-mon[33839]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:23.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:22 smithi072 ceph-mon[30177]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:23.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:22 smithi072 ceph-mon[41619]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:24.313 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:24 smithi173 systemd[1]: Stopping Ceph prometheus.a for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:03:24.614 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:24 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:03:24.337Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T02:03:24.615 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:24 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:03:24.337Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-26T02:03:24.615 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:24 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:03:24.337Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-26T02:03:24.615 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:24 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:03:24.337Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T02:03:24.615 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:24 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:03:24.337Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-26T02:03:24.615 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:24 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:03:24.337Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-26T02:03:24.615 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:24 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:03:24.337Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T02:03:24.615 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:24 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:03:24.337Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-26T02:03:24.615 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:24 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:03:24.337Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-26T02:03:24.615 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:24 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:03:24.337Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T02:03:24.615 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:24 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:03:24.337Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-26T02:03:24.615 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:24 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[64763]: ts=2024-03-26T02:03:24.337Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-26T02:03:24.615 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:24 smithi173 podman[68137]: 2024-03-26 02:03:24.347655376 +0000 UTC m=+0.131770460 container died d514371d5abc718f0fe6b9935e92d85352658bbc41a519523a7f97d14596c6b1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:03:24.890 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:24 smithi173 podman[68137]: 2024-03-26 02:03:24.713628938 +0000 UTC m=+0.497744072 container cleanup d514371d5abc718f0fe6b9935e92d85352658bbc41a519523a7f97d14596c6b1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:03:24.890 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:24 smithi173 bash[68137]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T02:03:25.218 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:24 smithi072 ceph-mon[30177]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:25.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:24 smithi072 ceph-mon[41619]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:24 smithi173 ceph-mon[33839]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:25.313 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 podman[68156]: 2024-03-26 02:03:25.007041547 +0000 UTC m=+0.662371352 container remove d514371d5abc718f0fe6b9935e92d85352658bbc41a519523a7f97d14596c6b1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:03:25.313 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 systemd[1]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-26T02:03:25.313 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 systemd[1]: Stopped Ceph prometheus.a for c86fbdd4-eb13-11ee-95d0-87774f69a715. 2024-03-26T02:03:25.313 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 systemd[1]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@prometheus.a.service: Consumed 1.151s CPU time. 2024-03-26T02:03:25.313 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 systemd[1]: Starting Ceph prometheus.a for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:03:25.606 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 podman[68256]: 2024-03-26 02:03:25.240973636 +0000 UTC m=+0.018263441 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-26T02:03:25.606 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 podman[68256]: 2024-03-26 02:03:25.357837686 +0000 UTC m=+0.135127472 container create 25ef588f85b82c172031eb42c1e78a51458adec36ca51a90656eb4eec527ada1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:03:25.606 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 podman[68256]: 2024-03-26 02:03:25.576691987 +0000 UTC m=+0.353981771 container init 25ef588f85b82c172031eb42c1e78a51458adec36ca51a90656eb4eec527ada1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:03:25.606 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 podman[68256]: 2024-03-26 02:03:25.582355075 +0000 UTC m=+0.359644858 container start 25ef588f85b82c172031eb42c1e78a51458adec36ca51a90656eb4eec527ada1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:03:26.063 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:03:25.606Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-26T02:03:26.064 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:03:25.606Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-26T02:03:26.064 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:03:25.606Z caller=main.go:570 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi173 (none))" 2024-03-26T02:03:26.064 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:03:25.606Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T02:03:26.064 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:03:25.606Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T02:03:26.064 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:03:25.608Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T02:03:26.064 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:03:25.608Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-26T02:03:26.064 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:03:25.610Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-26T02:03:26.064 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:03:25.610Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-26T02:03:26.064 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:03:25.611Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T02:03:26.064 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:03:25.611Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.582µs 2024-03-26T02:03:26.064 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:03:25.611Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T02:03:26.064 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:03:25.612Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-26T02:03:26.064 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:03:25.612Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-26T02:03:26.064 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:03:25.612Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=17.259µs wal_replay_duration=708.164µs wbl_replay_duration=131ns total_replay_duration=742.903µs 2024-03-26T02:03:26.065 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:03:25.613Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T02:03:26.065 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:03:25.613Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-26T02:03:26.065 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:03:25.613Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T02:03:26.065 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:03:25.631Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=18.127859ms db_storage=788ns remote_storage=1.109µs web_handler=276ns query_engine=580ns scrape=566.264µs scrape_sd=78.404µs notify=15.07µs notify_sd=15.548µs rules=17.243968ms tracing=6.08µs 2024-03-26T02:03:26.065 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:03:25.631Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-26T02:03:26.065 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:03:25.631Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-26T02:03:26.065 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 bash[68256]: 25ef588f85b82c172031eb42c1e78a51458adec36ca51a90656eb4eec527ada1 2024-03-26T02:03:26.065 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:03:25 smithi173 systemd[1]: Started Ceph prometheus.a for c86fbdd4-eb13-11ee-95d0-87774f69a715. 2024-03-26T02:03:26.110 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:03:25 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:03:25] ENGINE Bus STOPPING 2024-03-26T02:03:26.110 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:03:25 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:03:25] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T02:03:26.110 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:03:25 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:03:25] ENGINE Bus STOPPED 2024-03-26T02:03:26.110 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:03:25 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:03:25] ENGINE Bus STARTING 2024-03-26T02:03:26.110 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:03:25 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:03:25] ENGINE Serving on http://:::9283 2024-03-26T02:03:26.110 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:03:25 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:03:25] ENGINE Bus STARTED 2024-03-26T02:03:26.110 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:03:25 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:03:25] ENGINE Bus STOPPING 2024-03-26T02:03:26.111 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:03:25 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:03:25] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T02:03:26.111 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:03:25 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:03:25] ENGINE Bus STOPPED 2024-03-26T02:03:26.111 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:03:25 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:03:25] ENGINE Bus STARTING 2024-03-26T02:03:26.111 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:03:25 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:03:25] ENGINE Serving on http://:::9283 2024-03-26T02:03:26.111 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:03:25 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:03:25] ENGINE Bus STARTED 2024-03-26T02:03:26.111 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:03:25 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:03:25] ENGINE Bus STOPPING 2024-03-26T02:03:26.111 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:03:26] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T02:03:26.111 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:03:26] ENGINE Bus STOPPED 2024-03-26T02:03:26.111 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:03:26] ENGINE Bus STARTING 2024-03-26T02:03:26.111 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:03:26] ENGINE Serving on http://:::9283 2024-03-26T02:03:26.111 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:03:26] ENGINE Bus STARTED 2024-03-26T02:03:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:26 smithi173 ceph-mon[33839]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:27.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:26 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:27.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:26 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:27.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:26 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:03:27.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:26 smithi173 ceph-mon[33839]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:03:27.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:26 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi072.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T02:03:27.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:26 smithi173 ceph-mon[33839]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi072.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T02:03:27.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:26 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:27.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:26 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:03:27.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:26 smithi173 ceph-mon[33839]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:03:27.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:26 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi173.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T02:03:27.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:26 smithi173 ceph-mon[33839]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi173.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T02:03:27.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:26 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:27.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:26 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:03:27.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:26 smithi173 ceph-mon[33839]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:03:27.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:26 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi173.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T02:03:27.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:26 smithi173 ceph-mon[33839]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi173.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T02:03:27.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:26 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:27.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:26 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:03:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[30177]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:27.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:27.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:03:27.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[30177]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:03:27.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi072.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T02:03:27.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[30177]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi072.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T02:03:27.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:27.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:03:27.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[30177]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:03:27.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi173.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T02:03:27.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[30177]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi173.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T02:03:27.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:27.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:03:27.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[30177]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:03:27.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi173.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T02:03:27.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[30177]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi173.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T02:03:27.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:27.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:03:27.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[41619]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:27.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:27.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:27.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:03:27.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[41619]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T02:03:27.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi072.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T02:03:27.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[41619]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi072.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T02:03:27.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:27.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:03:27.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[41619]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T02:03:27.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi173.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T02:03:27.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[41619]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi173.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T02:03:27.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:27.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:03:27.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[41619]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T02:03:27.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi173.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T02:03:27.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[41619]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi173.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T02:03:27.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:27.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:26 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:03:28.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:28 smithi072 ceph-mon[30177]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:28.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:28 smithi072 ceph-mon[41619]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:28.969 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:03:28 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a[81552]: ts=2024-03-26T02:03:28.643Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003785851s 2024-03-26T02:03:29.005 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:28 smithi173 ceph-mon[33839]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:30.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:30 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:30.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:30 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:30.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:30 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:30.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:30 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:30.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:30 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:30.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:30 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:30.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:03:30 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:03:31.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:31 smithi072 ceph-mon[30177]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:31.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:31 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:31.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:31 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:31.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:31 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:03:31.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:31 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:03:31.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:31 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:31.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:31 smithi072 ceph-mon[41619]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:31.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:31 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:31.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:31 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:31.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:31 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:03:31.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:31 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:03:31.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:31 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:31.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:31 smithi173 ceph-mon[33839]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:31.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:31 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:31.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:31 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:31.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:31 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:03:31.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:31 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:03:31.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:31 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:03:32.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:32 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:03:32.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:32 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:03:32.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:32 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:03:33.468 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:33 smithi072 ceph-mon[30177]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:33.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:33 smithi072 ceph-mon[41619]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:33 smithi173 ceph-mon[33839]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:35.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:35 smithi072 ceph-mon[30177]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:03:35.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:35 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:03:35.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:03:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:03:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:03:35.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:35 smithi072 ceph-mon[41619]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:03:35.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:35 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:03:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:35 smithi173 ceph-mon[33839]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:03:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:35 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:03:37.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:37 smithi072 ceph-mon[30177]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:37.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:37 smithi072 ceph-mon[41619]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:37 smithi173 ceph-mon[33839]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:39.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:39 smithi072 ceph-mon[30177]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:03:39.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:39 smithi072 ceph-mon[41619]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:03:39.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:39 smithi173 ceph-mon[33839]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:03:40.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:03:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:03:41.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:41 smithi072 ceph-mon[30177]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:03:41.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:41 smithi072 ceph-mon[41619]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:03:41.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:41 smithi173 ceph-mon[33839]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:03:42.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:42 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:03:42.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:42 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:03:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:42 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:03:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:43 smithi072 ceph-mon[30177]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:43.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:43 smithi072 ceph-mon[41619]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:43 smithi173 ceph-mon[33839]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:45.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:45 smithi072 ceph-mon[30177]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:45.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:45 smithi072 ceph-mon[41619]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:45.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:03:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:03:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:03:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:45 smithi173 ceph-mon[33839]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:47.387 INFO:tasks.workunit.client.0.smithi072.stderr:Note: switching to '9f04c91b77e4619963d41730d8cf756e3a6152de'. 2024-03-26T02:03:47.388 INFO:tasks.workunit.client.0.smithi072.stderr: 2024-03-26T02:03:47.388 INFO:tasks.workunit.client.0.smithi072.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-26T02:03:47.388 INFO:tasks.workunit.client.0.smithi072.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-26T02:03:47.388 INFO:tasks.workunit.client.0.smithi072.stderr:state without impacting any branches by switching back to a branch. 2024-03-26T02:03:47.388 INFO:tasks.workunit.client.0.smithi072.stderr: 2024-03-26T02:03:47.388 INFO:tasks.workunit.client.0.smithi072.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-26T02:03:47.388 INFO:tasks.workunit.client.0.smithi072.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-26T02:03:47.388 INFO:tasks.workunit.client.0.smithi072.stderr: 2024-03-26T02:03:47.388 INFO:tasks.workunit.client.0.smithi072.stderr: git switch -c 2024-03-26T02:03:47.388 INFO:tasks.workunit.client.0.smithi072.stderr: 2024-03-26T02:03:47.388 INFO:tasks.workunit.client.0.smithi072.stderr:Or undo this operation with: 2024-03-26T02:03:47.388 INFO:tasks.workunit.client.0.smithi072.stderr: 2024-03-26T02:03:47.388 INFO:tasks.workunit.client.0.smithi072.stderr: git switch - 2024-03-26T02:03:47.389 INFO:tasks.workunit.client.0.smithi072.stderr: 2024-03-26T02:03:47.389 INFO:tasks.workunit.client.0.smithi072.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-26T02:03:47.389 INFO:tasks.workunit.client.0.smithi072.stderr: 2024-03-26T02:03:47.389 INFO:tasks.workunit.client.0.smithi072.stderr:HEAD is now at 9f04c91b77e Merge PR #56391 into squid 2024-03-26T02:03:47.392 DEBUG:teuthology.orchestra.run.smithi072:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-26T02:03:47.417 INFO:tasks.workunit.client.0.smithi072.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-26T02:03:47.419 INFO:tasks.workunit.client.0.smithi072.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-26T02:03:47.420 INFO:tasks.workunit.client.0.smithi072.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-26T02:03:47.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:47 smithi072 ceph-mon[30177]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:47.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:47 smithi072 ceph-mon[41619]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:47.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:47 smithi173 ceph-mon[33839]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:48.582 INFO:tasks.workunit.client.0.smithi072.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-26T02:03:48.689 INFO:tasks.workunit.client.0.smithi072.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-26T02:03:48.731 INFO:tasks.workunit.client.0.smithi072.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-26T02:03:48.732 INFO:tasks.workunit.client.0.smithi072.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-26T02:03:48.732 INFO:tasks.workunit.client.0.smithi072.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-26T02:03:48.772 INFO:tasks.workunit.client.0.smithi072.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-26T02:03:48.775 DEBUG:teuthology.orchestra.run.smithi072:> set -ex 2024-03-26T02:03:48.775 DEBUG:teuthology.orchestra.run.smithi072:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-26T02:03:48.831 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-03-26T02:03:48.832 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-03-26T02:03:48.833 DEBUG:teuthology.orchestra.run.smithi072:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9f04c91b77e4619963d41730d8cf756e3a6152de TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-03-26T02:03:49.005 INFO:tasks.workunit.client.0.smithi072.stderr:+ parallel=1 2024-03-26T02:03:49.005 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' '' = --serial ']' 2024-03-26T02:03:49.005 INFO:tasks.workunit.client.0.smithi072.stderr:+ crimson=0 2024-03-26T02:03:49.005 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' '' = --crimson ']' 2024-03-26T02:03:49.005 INFO:tasks.workunit.client.0.smithi072.stderr:+ color= 2024-03-26T02:03:49.005 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -t 1 ']' 2024-03-26T02:03:49.006 INFO:tasks.workunit.client.0.smithi072.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-03-26T02:03:49.006 INFO:tasks.workunit.client.0.smithi072.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-03-26T02:03:49.006 INFO:tasks.workunit.client.0.smithi072.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-03-26T02:03:49.007 INFO:tasks.workunit.client.0.smithi072.stderr:+ declare -A pids 2024-03-26T02:03:49.007 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.007 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.007 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s api_aio 2024-03-26T02:03:49.007 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' api_aio' 2024-03-26T02:03:49.007 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo api_aio 2024-03-26T02:03:49.007 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.008 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=api_aio 2024-03-26T02:03:49.008 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82318 2024-03-26T02:03:49.008 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test api_aio on pid 82318' 2024-03-26T02:03:49.008 INFO:tasks.workunit.client.0.smithi072.stdout:test api_aio on pid 82318 2024-03-26T02:03:49.012 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=82318 2024-03-26T02:03:49.012 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.012 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.012 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-03-26T02:03:49.013 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s api_aio_pp 2024-03-26T02:03:49.013 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' api_aio_pp' 2024-03-26T02:03:49.013 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo api_aio_pp 2024-03-26T02:03:49.013 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.013 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.013 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.013 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.013 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.013 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.013 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.013 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.013 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.013 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.013 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.014 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.014 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.014 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.014 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.014 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.014 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.014 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.014 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.014 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.014 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.014 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.015 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.015 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.015 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.015 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.015 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=api_aio_pp 2024-03-26T02:03:49.015 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82325 2024-03-26T02:03:49.015 INFO:tasks.workunit.client.0.smithi072.stdout:test api_aio_pp on pid 82325 2024-03-26T02:03:49.020 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test api_aio_pp on pid 82325' 2024-03-26T02:03:49.020 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=82325 2024-03-26T02:03:49.020 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.020 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.020 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-03-26T02:03:49.020 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s api_io 2024-03-26T02:03:49.020 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' api_io' 2024-03-26T02:03:49.020 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo api_io 2024-03-26T02:03:49.021 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.021 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.021 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.021 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.021 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.021 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.021 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.021 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.021 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.021 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.021 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.021 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.021 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.022 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.022 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.022 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.022 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.022 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.022 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.022 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.022 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.022 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.022 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.023 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.023 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.023 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.023 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.023 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.023 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.023 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.023 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.023 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.023 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.024 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.024 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.024 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.024 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.024 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.024 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.024 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.024 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.024 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.024 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.024 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.024 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.024 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.025 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=api_io 2024-03-26T02:03:49.025 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82334 2024-03-26T02:03:49.025 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test api_io on pid 82334' 2024-03-26T02:03:49.025 INFO:tasks.workunit.client.0.smithi072.stdout:test api_io on pid 82334 2024-03-26T02:03:49.030 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=82334 2024-03-26T02:03:49.030 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.030 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-03-26T02:03:49.030 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.030 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s api_io_pp 2024-03-26T02:03:49.031 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' api_io_pp' 2024-03-26T02:03:49.031 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo api_io_pp 2024-03-26T02:03:49.031 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.031 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.031 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.031 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.031 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.031 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.031 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.031 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.031 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.031 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.031 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.031 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.031 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.032 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.032 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.032 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.032 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.032 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.032 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.032 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.032 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.032 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.032 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.032 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.032 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.032 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.032 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.032 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.032 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.033 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.033 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.033 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.033 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.033 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.033 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.033 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.033 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.033 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.033 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.033 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.033 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.033 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.033 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.033 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.034 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.034 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.034 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.034 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.034 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.034 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.034 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.034 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.034 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.034 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.034 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.034 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=api_io_pp 2024-03-26T02:03:49.034 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82345 2024-03-26T02:03:49.035 INFO:tasks.workunit.client.0.smithi072.stdout:test api_io_pp on pid 82345 2024-03-26T02:03:49.040 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test api_io_pp on pid 82345' 2024-03-26T02:03:49.040 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=82345 2024-03-26T02:03:49.040 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.040 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-03-26T02:03:49.040 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.040 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s api_asio 2024-03-26T02:03:49.040 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' api_asio' 2024-03-26T02:03:49.040 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo api_asio 2024-03-26T02:03:49.040 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.041 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.041 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.041 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.041 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.041 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.041 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.041 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.041 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.041 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.041 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.041 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.041 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.041 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.041 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.041 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.041 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.042 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.042 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.042 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.042 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.042 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.042 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.042 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.042 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.042 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.042 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.042 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.042 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.042 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.042 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.042 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.043 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.043 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.043 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.043 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.043 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.043 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.043 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.043 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.043 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.043 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.043 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.043 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.043 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.043 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.043 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.044 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.044 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.044 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.044 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.044 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.044 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.044 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.044 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.044 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.044 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.044 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.044 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.044 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.044 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.044 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.045 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.045 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.045 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.045 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.045 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.045 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.045 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.045 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.045 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.045 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.045 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.045 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=api_asio 2024-03-26T02:03:49.045 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82360 2024-03-26T02:03:49.046 INFO:tasks.workunit.client.0.smithi072.stdout:test api_asio on pid 82360 2024-03-26T02:03:49.052 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test api_asio on pid 82360' 2024-03-26T02:03:49.052 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=82360 2024-03-26T02:03:49.052 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.052 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-03-26T02:03:49.053 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.053 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s api_list 2024-03-26T02:03:49.053 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.053 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.053 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' api_list' 2024-03-26T02:03:49.053 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.053 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.053 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.053 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.053 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo api_list 2024-03-26T02:03:49.053 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.053 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.053 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.053 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.054 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.054 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.054 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.054 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.054 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.054 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.054 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.054 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.054 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.054 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.054 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.054 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.054 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.054 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.054 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.055 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.055 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.055 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.055 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.055 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.055 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.055 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.055 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.055 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.055 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.055 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.055 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.055 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.055 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.055 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.055 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.056 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.056 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.056 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.056 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.056 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.056 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.056 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.056 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.056 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.056 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.056 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.056 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.056 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.056 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.056 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.057 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.057 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.057 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.057 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.057 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.057 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.057 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.057 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.057 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.057 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.057 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.057 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.057 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.057 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.057 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.057 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.057 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.058 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.058 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.058 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.058 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.058 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.058 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.058 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.058 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.058 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.058 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.058 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.058 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.058 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.058 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.059 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.059 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.059 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.059 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.059 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.059 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.059 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.059 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.059 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.059 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.059 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.059 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.059 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.059 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.059 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.060 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.060 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.060 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.060 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/82318/exe 2024-03-26T02:03:49.060 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=api_list 2024-03-26T02:03:49.060 INFO:tasks.workunit.client.0.smithi072.stdout:test api_list on pid 82377 2024-03-26T02:03:49.065 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82377 2024-03-26T02:03:49.065 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.065 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test api_list on pid 82377' 2024-03-26T02:03:49.065 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=82377 2024-03-26T02:03:49.065 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.065 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.065 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-03-26T02:03:49.066 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s api_lock 2024-03-26T02:03:49.066 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' api_lock' 2024-03-26T02:03:49.066 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.066 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo api_lock 2024-03-26T02:03:49.066 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.066 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.066 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.066 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.066 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.066 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.066 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.066 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.066 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.066 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.067 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.067 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.067 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.067 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.067 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.067 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.067 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.067 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.067 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.067 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.067 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.067 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.067 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.067 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.067 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.068 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.068 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.068 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.068 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.068 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.068 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.068 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.068 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.068 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.068 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.068 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.068 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.068 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.068 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.068 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.068 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.069 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.069 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.069 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.069 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.069 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.069 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.069 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.069 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.069 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.069 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.069 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.069 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.069 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.069 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.069 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.070 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.070 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.070 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.070 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.070 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.070 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.070 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.070 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.070 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.070 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.070 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.070 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.070 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.070 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.070 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.071 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.071 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.071 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.071 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.071 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.071 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.071 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.071 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.071 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.071 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-03-26T02:03:49.071 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.071 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_api_aio.log 2024-03-26T02:03:49.071 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.071 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/82325/exe 2024-03-26T02:03:49.072 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.072 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ api_aio: /' 2024-03-26T02:03:49.072 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.072 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.072 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.072 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.072 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.072 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.072 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.072 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.072 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.072 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.072 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=api_lock 2024-03-26T02:03:49.072 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.072 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.073 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.073 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.073 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.073 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.073 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.073 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.073 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.073 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82397 2024-03-26T02:03:49.073 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.073 INFO:tasks.workunit.client.0.smithi072.stdout:test api_lock on pid 82397 2024-03-26T02:03:49.077 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.077 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test api_lock on pid 82397' 2024-03-26T02:03:49.077 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=82397 2024-03-26T02:03:49.077 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.077 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.077 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.077 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s api_lock_pp 2024-03-26T02:03:49.077 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-03-26T02:03:49.077 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.078 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.078 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.078 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' api_lock_pp' 2024-03-26T02:03:49.078 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.078 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.078 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.078 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.078 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.078 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.078 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.078 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.078 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.078 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.078 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.079 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.079 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.079 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.079 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.079 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.079 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.079 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.079 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.079 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.079 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.079 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.079 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.079 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.079 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.080 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.080 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.080 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.080 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.080 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.080 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.080 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.080 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.080 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.080 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.080 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.080 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.080 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.080 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.080 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.081 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ api_aio_pp: /' 2024-03-26T02:03:49.081 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo api_lock_pp 2024-03-26T02:03:49.081 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.081 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-03-26T02:03:49.081 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.081 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.081 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.081 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.081 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.081 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.081 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-03-26T02:03:49.081 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.081 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.081 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.081 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.081 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.082 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.082 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.082 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.082 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.082 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.082 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.082 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.082 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.082 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.082 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.082 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.082 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.082 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.082 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.082 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.083 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.083 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.083 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.083 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.083 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.083 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.083 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.083 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.083 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.083 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.083 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.083 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.083 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.083 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.083 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.084 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.084 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.084 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.084 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.084 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.084 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.084 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.084 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.084 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.084 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.084 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.084 INFO:tasks.workunit.client.0.smithi072.stdout:test api_lock_pp on pid 82417 2024-03-26T02:03:49.088 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=api_lock_pp 2024-03-26T02:03:49.088 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82417 2024-03-26T02:03:49.088 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test api_lock_pp on pid 82417' 2024-03-26T02:03:49.088 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=82417 2024-03-26T02:03:49.088 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.088 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.088 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/82334/exe 2024-03-26T02:03:49.088 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-03-26T02:03:49.088 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.088 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.088 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.088 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.089 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.089 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.089 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.089 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.089 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.089 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.089 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.089 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.089 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s api_misc 2024-03-26T02:03:49.089 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.089 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.089 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.089 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.089 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.089 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.090 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.090 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.090 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.090 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.090 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.090 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.090 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.090 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.090 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.090 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.090 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' api_misc' 2024-03-26T02:03:49.090 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.090 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.091 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.091 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.091 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.091 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.091 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.091 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.091 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.091 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.091 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.091 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.091 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.091 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.091 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.091 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.092 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.092 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.092 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.092 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.092 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.092 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.092 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.092 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.092 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.092 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.092 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.092 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.092 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.092 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.092 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.093 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.093 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.093 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.093 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/82345/exe 2024-03-26T02:03:49.093 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.093 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.093 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo api_misc 2024-03-26T02:03:49.093 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.093 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.093 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.093 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.093 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.093 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.093 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.093 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.094 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.094 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.094 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.094 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.094 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.094 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.094 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.094 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.094 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.094 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.094 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.094 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.094 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.094 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.094 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.095 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.095 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.095 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.095 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.095 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.095 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.095 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.095 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.095 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.095 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.095 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.095 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.095 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.095 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.095 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.095 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ api_io: /' 2024-03-26T02:03:49.096 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.096 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.096 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.096 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.096 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.096 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.096 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.096 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.096 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.096 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.096 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.096 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.096 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.096 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.096 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.096 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.096 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.097 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.097 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.097 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.097 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.097 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.097 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.097 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.097 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.097 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.097 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.097 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.097 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.097 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.097 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.097 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.097 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.097 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.098 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.098 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.098 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.098 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.098 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.098 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.098 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-03-26T02:03:49.098 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-03-26T02:03:49.098 INFO:tasks.workunit.client.0.smithi072.stdout:test api_misc on pid 82441 2024-03-26T02:03:49.101 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=api_misc 2024-03-26T02:03:49.101 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82441 2024-03-26T02:03:49.101 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test api_misc on pid 82441' 2024-03-26T02:03:49.101 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=82441 2024-03-26T02:03:49.101 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.101 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.101 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.101 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.101 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.101 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.101 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.101 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.101 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.101 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.101 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.102 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.102 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.102 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.102 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.102 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.102 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.102 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.102 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.102 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.102 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.102 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.102 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.102 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.102 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.103 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.103 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.103 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ api_io_pp: /' 2024-03-26T02:03:49.103 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.103 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.103 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.103 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.103 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.103 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.103 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.103 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.103 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.103 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.103 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.103 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.103 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.104 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.104 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.104 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s api_misc_pp 2024-03-26T02:03:49.104 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.104 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-03-26T02:03:49.104 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' api_misc_pp' 2024-03-26T02:03:49.104 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/82360/exe 2024-03-26T02:03:49.104 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo api_misc_pp 2024-03-26T02:03:49.104 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.104 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-03-26T02:03:49.104 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.104 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.104 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.104 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.104 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.105 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.105 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.105 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.105 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.105 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.105 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.105 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.105 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.105 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.105 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.105 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.105 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.105 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.105 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.105 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.106 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.106 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.106 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.106 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.106 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.106 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.106 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.106 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.106 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.106 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.106 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.106 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.106 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.106 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.106 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.107 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.107 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.107 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.107 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.107 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.107 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.107 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.107 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.107 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.107 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.107 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.107 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.107 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.107 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.107 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.107 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.108 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.108 INFO:tasks.workunit.client.0.smithi072.stdout:test api_misc_pp on pid 82459 2024-03-26T02:03:49.111 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.111 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.111 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.111 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.111 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.111 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.111 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.111 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.112 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.112 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.112 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=api_misc_pp 2024-03-26T02:03:49.112 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82459 2024-03-26T02:03:49.112 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test api_misc_pp on pid 82459' 2024-03-26T02:03:49.112 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=82459 2024-03-26T02:03:49.112 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.112 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.112 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s api_tier_pp 2024-03-26T02:03:49.112 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ api_asio: /' 2024-03-26T02:03:49.112 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-03-26T02:03:49.112 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' api_tier_pp' 2024-03-26T02:03:49.112 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_api_io.log 2024-03-26T02:03:49.112 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.112 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.112 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.112 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.113 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.113 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.113 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.113 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.113 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.113 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.113 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.113 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.113 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.113 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.113 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.113 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.113 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.113 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.113 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.113 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.114 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.114 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.114 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.114 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-03-26T02:03:49.114 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.114 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.114 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.114 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.114 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.114 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.114 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.114 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.114 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.114 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.114 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo api_tier_pp 2024-03-26T02:03:49.115 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.115 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.115 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.115 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.115 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.115 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.115 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.115 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.115 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.115 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.115 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.115 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.115 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.115 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.115 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.116 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.116 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.116 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.116 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.116 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.116 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.116 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.116 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.116 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.116 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.116 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.116 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.116 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.116 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.116 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.117 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.117 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.117 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.117 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.117 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.117 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.117 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.117 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.117 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.117 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.117 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.117 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.117 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.117 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.117 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.117 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.118 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.118 INFO:tasks.workunit.client.0.smithi072.stdout:test api_tier_pp on pid 82476 2024-03-26T02:03:49.120 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=api_tier_pp 2024-03-26T02:03:49.120 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82476 2024-03-26T02:03:49.120 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test api_tier_pp on pid 82476' 2024-03-26T02:03:49.120 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=82476 2024-03-26T02:03:49.121 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.121 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.121 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s api_pool 2024-03-26T02:03:49.121 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' api_pool' 2024-03-26T02:03:49.121 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.121 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.121 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.121 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.121 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.121 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-03-26T02:03:49.121 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.121 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.121 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.122 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.122 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.122 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.122 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.122 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.122 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.122 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.122 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.122 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.122 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.122 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.122 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.122 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.122 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.122 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.123 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.123 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.123 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.123 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.123 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.123 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.123 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.123 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.123 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.123 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.123 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/82377/exe 2024-03-26T02:03:49.123 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.123 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.123 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.123 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.124 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.124 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.124 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.124 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.124 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.124 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.124 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.124 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.124 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.124 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.124 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.124 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.124 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.124 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.124 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.125 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.125 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.125 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.125 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.125 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.125 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.125 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.125 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.125 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.125 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.125 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.125 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.125 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.125 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.125 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.125 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.126 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.126 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.126 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.126 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.126 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.126 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.126 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.126 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo api_pool 2024-03-26T02:03:49.126 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.126 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.126 INFO:tasks.workunit.client.0.smithi072.stdout:test api_pool on pid 82489 2024-03-26T02:03:49.132 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.132 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.132 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_api_asio.log 2024-03-26T02:03:49.132 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/82417/exe 2024-03-26T02:03:49.132 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.132 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.132 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.132 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.132 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=api_pool 2024-03-26T02:03:49.132 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.132 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.132 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.132 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.132 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.132 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.133 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.133 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.133 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.133 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.133 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.133 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82489 2024-03-26T02:03:49.133 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.133 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test api_pool on pid 82489' 2024-03-26T02:03:49.133 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.133 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.133 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=82489 2024-03-26T02:03:49.133 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.133 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.133 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.133 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.134 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.134 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.134 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.134 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.134 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.134 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.134 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.134 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.134 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.134 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.134 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.134 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.134 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.135 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.135 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ api_list: /' 2024-03-26T02:03:49.135 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.135 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-03-26T02:03:49.135 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.135 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.135 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-03-26T02:03:49.135 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_api_list.log 2024-03-26T02:03:49.135 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-03-26T02:03:49.135 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ api_lock_pp: /' 2024-03-26T02:03:49.135 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.135 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.135 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.135 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.136 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.136 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.136 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-03-26T02:03:49.136 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.136 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.136 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.136 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.136 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.136 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.136 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.136 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.136 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.136 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.136 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.137 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.137 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.137 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.137 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.137 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.137 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.137 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.137 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.137 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.137 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.137 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.137 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.137 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.137 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.138 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.138 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.138 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.138 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.138 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.138 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.138 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.138 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.138 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.138 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.138 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.138 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.138 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.138 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.138 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.139 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.139 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.139 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.139 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.139 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.139 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.139 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.139 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.139 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.139 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.139 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.139 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.139 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.139 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.140 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.140 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.140 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.140 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.140 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.140 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.140 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s api_snapshots 2024-03-26T02:03:49.140 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.140 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.140 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.140 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' api_snapshots' 2024-03-26T02:03:49.140 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.140 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.140 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.140 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.140 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.140 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.141 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.141 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.141 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.141 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.141 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.141 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.141 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.141 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.141 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.141 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.141 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.141 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.141 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.141 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.141 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.142 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.142 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.142 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.142 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.142 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.142 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.142 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.142 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.142 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.142 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.142 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.142 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.142 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.142 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.142 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.143 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.143 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.143 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.143 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.143 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.143 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.143 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.143 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.143 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.143 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.143 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.143 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.143 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.143 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.143 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.144 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.144 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.144 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.144 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.144 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.144 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.144 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.144 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.144 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.144 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.144 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.144 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.144 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.144 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.144 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.145 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.145 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.145 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.145 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.145 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.145 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.145 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/82441/exe 2024-03-26T02:03:49.145 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.145 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo api_snapshots 2024-03-26T02:03:49.145 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.145 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.145 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.145 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.145 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.145 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.145 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.146 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.146 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.146 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.146 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.146 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.146 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.146 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.146 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.146 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.146 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.146 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.146 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.146 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.147 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.147 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.147 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.147 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.147 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.147 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.147 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.147 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.147 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.147 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.147 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.147 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_api_misc.log 2024-03-26T02:03:49.147 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ api_misc: /' 2024-03-26T02:03:49.147 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-03-26T02:03:49.148 INFO:tasks.workunit.client.0.smithi072.stdout:test api_snapshots on pid 82575 2024-03-26T02:03:49.151 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=api_snapshots 2024-03-26T02:03:49.151 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82575 2024-03-26T02:03:49.151 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test api_snapshots on pid 82575' 2024-03-26T02:03:49.151 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=82575 2024-03-26T02:03:49.151 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.151 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.151 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.151 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.151 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.151 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.151 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.151 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.152 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.152 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s api_snapshots_pp 2024-03-26T02:03:49.152 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.152 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.152 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.152 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.152 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.152 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' api_snapshots_pp' 2024-03-26T02:03:49.152 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.152 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.152 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.152 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.152 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.152 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.152 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.153 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.153 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.153 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.153 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.153 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-03-26T02:03:49.153 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.153 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.153 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.153 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.153 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.153 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.153 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.153 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.153 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.153 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.154 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.154 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.154 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.154 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.154 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.154 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.154 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.154 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.154 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.154 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.154 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.154 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.154 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.154 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.154 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.155 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.155 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.155 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.155 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.155 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.155 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.155 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.155 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/82397/exe 2024-03-26T02:03:49.155 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo api_snapshots_pp 2024-03-26T02:03:49.155 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.155 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.155 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.155 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.155 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.156 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.156 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.156 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.156 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.156 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.156 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.156 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.156 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.156 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.156 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.156 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.156 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.156 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.156 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.156 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.157 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.157 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.157 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.157 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_api_lock.log 2024-03-26T02:03:49.157 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.157 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.157 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.157 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.157 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.157 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.157 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.157 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.157 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.157 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.157 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.158 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.158 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.158 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.158 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.158 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.158 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.158 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.158 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.158 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.158 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.158 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.158 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.158 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.158 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.158 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ api_lock: /' 2024-03-26T02:03:49.159 INFO:tasks.workunit.client.0.smithi072.stdout:test api_snapshots_pp on pid 82611 2024-03-26T02:03:49.160 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=api_snapshots_pp 2024-03-26T02:03:49.160 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82611 2024-03-26T02:03:49.160 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test api_snapshots_pp on pid 82611' 2024-03-26T02:03:49.160 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=82611 2024-03-26T02:03:49.160 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.161 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.161 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s api_stat 2024-03-26T02:03:49.161 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-03-26T02:03:49.161 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.161 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.161 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.161 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.161 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.161 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.161 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.161 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.161 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.161 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' api_stat' 2024-03-26T02:03:49.161 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.161 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.162 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.162 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.162 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.162 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.162 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.162 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.162 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.162 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.162 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.162 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.162 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.162 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.162 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.162 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.163 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.163 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.163 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.163 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.163 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.163 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.163 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.163 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.163 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.163 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.163 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.163 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.163 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.163 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.163 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.163 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo api_stat 2024-03-26T02:03:49.164 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/82459/exe 2024-03-26T02:03:49.164 INFO:tasks.workunit.client.0.smithi072.stdout:test api_stat on pid 82624 2024-03-26T02:03:49.170 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=api_stat 2024-03-26T02:03:49.170 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82624 2024-03-26T02:03:49.171 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test api_stat on pid 82624' 2024-03-26T02:03:49.171 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=82624 2024-03-26T02:03:49.171 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.171 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.171 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.171 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s api_stat_pp 2024-03-26T02:03:49.171 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-03-26T02:03:49.171 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' api_stat_pp' 2024-03-26T02:03:49.171 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-03-26T02:03:49.171 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.171 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.171 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.171 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.171 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.171 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.172 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.172 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.172 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.172 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.172 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.172 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.172 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.172 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.172 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.172 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.172 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.172 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.172 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.172 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.173 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.173 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.173 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.173 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.173 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.173 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.173 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.173 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.173 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.173 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.173 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.173 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.173 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.173 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.173 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.173 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.174 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.174 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.174 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.174 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.174 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.174 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.174 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.174 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.174 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-03-26T02:03:49.174 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.174 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.174 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.174 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.174 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.174 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.175 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ api_misc_pp: /' 2024-03-26T02:03:49.175 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-03-26T02:03:49.175 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/82489/exe 2024-03-26T02:03:49.175 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.175 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.175 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.175 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.175 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.175 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.175 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.175 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.175 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.175 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.175 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.175 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.175 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.176 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.176 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.176 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.176 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.176 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.176 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.176 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.176 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.176 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.176 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.176 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.176 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.176 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.176 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.176 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.176 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.177 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.177 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.177 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.177 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.177 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.177 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.177 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.177 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.177 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.177 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.177 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ api_pool: /' 2024-03-26T02:03:49.177 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.177 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.177 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.177 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.177 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.178 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.178 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.178 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.178 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.178 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.178 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.178 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.178 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.178 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.178 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.178 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.178 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.178 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_api_pool.log 2024-03-26T02:03:49.178 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo api_stat_pp 2024-03-26T02:03:49.178 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-03-26T02:03:49.179 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.179 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.179 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.179 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.179 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.179 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.179 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.179 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.179 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.179 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.179 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.179 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.179 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.179 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.179 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.180 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.180 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.180 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.180 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.180 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.180 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.180 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.180 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.180 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.180 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.180 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.180 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.180 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.180 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.180 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.180 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.181 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.181 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.181 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.181 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.181 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.181 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.181 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.181 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.181 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.181 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.181 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.181 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.181 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.181 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.181 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.182 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.182 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.182 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.182 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.182 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.182 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.182 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.182 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.182 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.182 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.182 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.182 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.182 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.182 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.182 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.183 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.183 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.183 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.183 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.183 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.183 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.183 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.183 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.183 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.183 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.183 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.183 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.183 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.183 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.183 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.184 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.184 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.184 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.184 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.184 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.184 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.184 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.184 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.184 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.184 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.184 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.184 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.184 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.184 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.184 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.185 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.185 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.185 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.185 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.185 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.185 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.185 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.185 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.185 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.185 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.185 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.185 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.185 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.185 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.185 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.186 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.186 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.186 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.186 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.186 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.186 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.186 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.186 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.186 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.186 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.186 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.186 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.186 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.186 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.186 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.187 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/82476/exe 2024-03-26T02:03:49.187 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.187 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.187 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.187 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.187 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.187 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.187 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.187 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.187 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.187 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.187 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.187 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.187 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.187 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.188 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.188 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.188 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.188 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.188 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.188 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.188 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.188 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.188 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.188 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.188 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.188 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.188 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.188 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.188 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.188 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.189 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ api_tier_pp: /' 2024-03-26T02:03:49.189 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.189 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.189 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.189 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.189 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.189 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.189 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.189 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.189 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.189 INFO:tasks.workunit.client.0.smithi072.stdout:test api_stat_pp on pid 82696 2024-03-26T02:03:49.196 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=api_stat_pp 2024-03-26T02:03:49.196 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82696 2024-03-26T02:03:49.196 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test api_stat_pp on pid 82696' 2024-03-26T02:03:49.196 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=82696 2024-03-26T02:03:49.196 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.196 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.196 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.196 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.197 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.197 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.197 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.197 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.197 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.197 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.197 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.197 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.197 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.197 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.197 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.197 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.197 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.197 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.197 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.198 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.198 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.198 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.198 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.198 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.198 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.198 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.198 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.198 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.198 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.198 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.198 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.198 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.198 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.198 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.199 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.199 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.199 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.199 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.199 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.199 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.199 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.199 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.199 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.199 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.199 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.199 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.199 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.199 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/82611/exe 2024-03-26T02:03:49.199 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.199 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.200 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.200 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.200 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.200 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-03-26T02:03:49.200 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.200 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.200 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.200 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.200 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.200 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.200 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.200 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.200 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.200 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.200 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.200 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.201 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.201 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.201 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.201 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.201 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.201 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-03-26T02:03:49.201 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-03-26T02:03:49.201 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-03-26T02:03:49.201 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.201 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s api_watch_notify 2024-03-26T02:03:49.201 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' api_watch_notify' 2024-03-26T02:03:49.201 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.201 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.201 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.201 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.202 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.202 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.202 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.202 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.202 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.202 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.202 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.202 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.202 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.202 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.202 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.202 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.202 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.202 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.202 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.203 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.203 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.203 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.203 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.203 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.203 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.203 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.203 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.203 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.203 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.203 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.203 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.203 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.203 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.203 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.203 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.204 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.204 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.204 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.204 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.204 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.204 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.204 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.204 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-03-26T02:03:49.204 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/82624/exe 2024-03-26T02:03:49.204 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.204 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.204 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.204 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.204 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.204 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.205 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo api_watch_notify 2024-03-26T02:03:49.205 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.205 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.205 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/82575/exe 2024-03-26T02:03:49.205 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.205 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.205 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.205 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.205 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.205 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.205 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.205 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.205 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.205 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.205 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.206 INFO:tasks.workunit.client.0.smithi072.stdout:test api_watch_notify on pid 82726 2024-03-26T02:03:49.208 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=api_watch_notify 2024-03-26T02:03:49.209 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.209 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.209 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.209 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.209 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.209 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82726 2024-03-26T02:03:49.209 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.209 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test api_watch_notify on pid 82726' 2024-03-26T02:03:49.209 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=82726 2024-03-26T02:03:49.209 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.209 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.209 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.209 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.210 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.210 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.210 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s api_watch_notify_pp 2024-03-26T02:03:49.210 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' api_watch_notify_pp' 2024-03-26T02:03:49.210 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-03-26T02:03:49.210 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-03-26T02:03:49.210 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_api_stat.log 2024-03-26T02:03:49.210 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-03-26T02:03:49.210 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.210 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-03-26T02:03:49.210 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.210 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.210 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.210 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.210 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.210 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ api_stat: /' 2024-03-26T02:03:49.211 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.211 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.211 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.211 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.211 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.211 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.211 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.211 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.211 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.211 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.211 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.211 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.211 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.211 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.211 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.212 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.212 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.212 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.212 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.212 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.212 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.212 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.212 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.212 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.212 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.212 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.212 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.212 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.212 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.213 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.213 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.213 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.213 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.213 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.213 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.213 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.213 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.213 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.213 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo api_watch_notify_pp 2024-03-26T02:03:49.213 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.213 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-03-26T02:03:49.213 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ api_snapshots: /' 2024-03-26T02:03:49.214 INFO:tasks.workunit.client.0.smithi072.stdout:test api_watch_notify_pp on pid 82783 2024-03-26T02:03:49.215 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=api_watch_notify_pp 2024-03-26T02:03:49.215 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.215 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82783 2024-03-26T02:03:49.215 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test api_watch_notify_pp on pid 82783' 2024-03-26T02:03:49.216 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.216 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=82783 2024-03-26T02:03:49.216 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.216 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.216 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.216 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.216 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.216 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.216 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.216 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.216 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.216 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.216 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.216 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.216 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.217 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.217 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.217 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.217 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.217 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.217 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.217 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.217 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-03-26T02:03:49.217 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.217 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.217 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.217 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.217 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.217 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.217 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.217 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.218 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.218 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.218 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.218 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.218 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.218 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.218 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.218 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.218 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.218 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.218 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.218 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.218 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.218 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.218 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.219 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.219 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.219 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.219 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s api_cmd 2024-03-26T02:03:49.219 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' api_cmd' 2024-03-26T02:03:49.219 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.219 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.219 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.219 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.219 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.219 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.219 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.219 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.219 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.219 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.220 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo api_cmd 2024-03-26T02:03:49.220 INFO:tasks.workunit.client.0.smithi072.stdout:test api_cmd on pid 82808 2024-03-26T02:03:49.224 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=api_cmd 2024-03-26T02:03:49.224 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82808 2024-03-26T02:03:49.224 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test api_cmd on pid 82808' 2024-03-26T02:03:49.224 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=82808 2024-03-26T02:03:49.224 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.224 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.224 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.224 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.224 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.224 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.224 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.224 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.224 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.225 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.225 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.225 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s api_cmd_pp 2024-03-26T02:03:49.225 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' api_cmd_pp' 2024-03-26T02:03:49.225 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.225 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.225 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.225 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.225 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.225 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.225 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.225 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.225 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.225 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.225 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.226 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.226 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.226 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.226 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.226 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.226 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.226 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.226 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.226 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.226 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.226 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.226 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.226 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.226 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.227 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.227 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.227 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.227 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.227 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.227 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.227 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.227 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.227 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.227 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.227 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.227 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.227 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.227 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.227 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.227 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.228 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.228 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.228 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.228 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.228 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.228 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.228 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.228 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.228 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.228 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.228 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.228 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.228 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.228 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.228 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.229 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.229 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.229 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.229 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.229 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.229 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.229 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.229 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.229 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.229 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.229 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-03-26T02:03:49.229 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.229 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.229 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.229 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.230 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.230 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.230 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.230 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.230 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.230 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.230 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.230 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.230 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.230 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.230 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.230 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.230 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.230 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.230 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.231 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.231 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.231 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.231 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.231 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.231 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.231 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.231 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.231 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.231 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.231 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.231 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.231 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.231 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.231 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.232 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.232 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.232 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.232 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.232 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.232 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.232 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.232 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.232 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.232 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.232 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.232 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.232 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.232 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.232 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo api_cmd_pp 2024-03-26T02:03:49.232 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.233 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.233 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.233 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.233 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.233 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.233 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.233 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.233 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.233 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.233 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.233 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.233 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.233 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.233 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.234 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.234 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.234 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.234 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.234 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.234 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.234 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/82696/exe 2024-03-26T02:03:49.234 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.234 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.234 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.234 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.234 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.234 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.234 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.234 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.235 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.235 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.235 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.235 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.235 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.235 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.235 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.235 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.235 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.235 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.235 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.235 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.235 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.235 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=api_cmd_pp 2024-03-26T02:03:49.236 INFO:tasks.workunit.client.0.smithi072.stdout:test api_cmd_pp on pid 82870 2024-03-26T02:03:49.240 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82870 2024-03-26T02:03:49.240 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test api_cmd_pp on pid 82870' 2024-03-26T02:03:49.240 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=82870 2024-03-26T02:03:49.240 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.240 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.240 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-03-26T02:03:49.240 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.240 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.240 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.240 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.240 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.241 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.241 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/82726/exe 2024-03-26T02:03:49.241 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.241 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s api_service 2024-03-26T02:03:49.241 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.241 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.241 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.241 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' api_service' 2024-03-26T02:03:49.241 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.241 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.241 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.241 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ api_stat_pp: /' 2024-03-26T02:03:49.241 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.241 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.241 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.242 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.242 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.242 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.242 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.242 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.242 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.242 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.242 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.242 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.242 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.242 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.242 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.242 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.242 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo api_service 2024-03-26T02:03:49.242 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.243 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.243 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.243 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.243 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.243 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.243 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.243 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-03-26T02:03:49.243 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-03-26T02:03:49.243 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.243 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.243 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.243 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.243 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.243 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.243 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.243 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.244 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.244 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.244 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.244 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.244 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.244 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.244 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.244 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.244 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.244 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.244 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.244 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.244 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.244 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.244 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.245 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.245 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.245 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.245 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.245 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.245 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.245 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.245 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.245 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.245 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.245 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.245 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.245 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.245 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.245 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.245 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.246 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ api_watch_notify: /' 2024-03-26T02:03:49.246 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.246 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-03-26T02:03:49.246 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-03-26T02:03:49.246 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.246 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.246 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.246 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.246 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.246 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.246 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.246 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.246 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.246 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.246 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.247 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.247 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.247 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.247 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.247 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.247 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.247 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.247 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.247 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.247 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.247 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.247 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.247 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.247 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.247 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.247 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.248 INFO:tasks.workunit.client.0.smithi072.stdout:test api_service on pid 82923 2024-03-26T02:03:49.252 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=api_service 2024-03-26T02:03:49.252 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82923 2024-03-26T02:03:49.252 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test api_service on pid 82923' 2024-03-26T02:03:49.252 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=82923 2024-03-26T02:03:49.252 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.252 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.252 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-03-26T02:03:49.252 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.252 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.252 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.252 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.252 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.252 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.253 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.253 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.253 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.253 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.253 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.253 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.253 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.253 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.253 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.253 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.253 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.253 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.253 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.253 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.253 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.254 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.254 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.254 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.254 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.254 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.254 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.254 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.254 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.254 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.254 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.254 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.254 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.254 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.254 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.254 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.255 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.255 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.255 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.255 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.255 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.255 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.255 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.255 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.255 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.255 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.255 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.255 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.255 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.255 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.255 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.256 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.256 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.256 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.256 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.256 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.256 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.256 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.256 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.256 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.256 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.256 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.256 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.256 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.256 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.256 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.256 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.257 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.257 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.257 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.257 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/82783/exe 2024-03-26T02:03:49.257 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.257 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.257 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.257 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.257 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.257 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.257 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.257 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.257 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.257 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.257 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.257 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.258 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.258 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.258 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.258 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.258 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.258 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.258 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.258 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.258 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.258 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.258 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.258 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.258 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.258 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.258 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.259 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.259 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.259 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.259 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.259 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.259 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.259 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.259 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.259 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.259 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.259 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.259 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.259 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.259 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.259 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.260 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.260 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.260 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.260 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.260 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-03-26T02:03:49.260 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.260 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.260 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.260 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.260 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.260 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.260 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.260 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.260 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.260 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.260 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.261 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.261 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.261 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.261 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.261 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.261 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.261 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.261 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.261 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.261 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.261 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.261 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.261 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.261 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.261 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.262 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.262 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.262 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.262 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.262 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.262 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.262 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.262 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.262 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.262 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.262 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.262 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.262 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.262 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.262 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.263 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.263 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.263 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.263 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.263 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.263 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-03-26T02:03:49.263 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.263 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.263 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.263 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.263 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.263 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.263 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-03-26T02:03:49.263 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.263 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.263 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.264 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.264 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.264 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.264 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.264 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s api_service_pp 2024-03-26T02:03:49.264 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' api_service_pp' 2024-03-26T02:03:49.264 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/82808/exe 2024-03-26T02:03:49.264 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.264 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.264 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.264 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.264 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo api_service_pp 2024-03-26T02:03:49.264 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.264 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.264 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.265 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.265 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.265 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.265 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.265 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.265 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.265 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.265 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.265 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.265 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.265 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.265 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.265 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.265 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.265 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.265 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.266 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.266 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.266 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.266 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.266 INFO:tasks.workunit.client.0.smithi072.stdout:test api_service_pp on pid 82962 2024-03-26T02:03:49.268 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=api_service_pp 2024-03-26T02:03:49.269 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ api_cmd: /' 2024-03-26T02:03:49.269 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82962 2024-03-26T02:03:49.269 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test api_service_pp on pid 82962' 2024-03-26T02:03:49.269 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=82962 2024-03-26T02:03:49.269 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.269 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.269 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_api_cmd.log 2024-03-26T02:03:49.269 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.269 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.269 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.269 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.269 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.269 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.269 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.270 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.270 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.270 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.270 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.270 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.270 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.270 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.270 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.270 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.270 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.270 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.270 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.270 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.270 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.270 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s api_c_write_operations 2024-03-26T02:03:49.271 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' api_c_write_operations' 2024-03-26T02:03:49.271 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-03-26T02:03:49.271 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-03-26T02:03:49.271 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/82870/exe 2024-03-26T02:03:49.271 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.271 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.271 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.271 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.271 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.271 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.271 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.271 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.271 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.271 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.271 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.271 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.272 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.272 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.272 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.272 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.272 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.272 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.272 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo api_c_write_operations 2024-03-26T02:03:49.272 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.272 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.272 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.272 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.272 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.272 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.272 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.272 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.272 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.273 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.273 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.273 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.273 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.273 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.273 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.273 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.273 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.273 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.273 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.273 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.273 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.273 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.273 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.273 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-03-26T02:03:49.274 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-03-26T02:03:49.274 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.274 INFO:tasks.workunit.client.0.smithi072.stdout:test api_c_write_operations on pid 82984 2024-03-26T02:03:49.278 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=api_c_write_operations 2024-03-26T02:03:49.278 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82984 2024-03-26T02:03:49.278 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test api_c_write_operations on pid 82984' 2024-03-26T02:03:49.278 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=82984 2024-03-26T02:03:49.278 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.278 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.278 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-03-26T02:03:49.278 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.278 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.278 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-03-26T02:03:49.279 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.279 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.279 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.279 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.279 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.279 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.279 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.279 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.279 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.279 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.279 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.279 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.279 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.279 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.279 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.280 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.280 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.280 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.280 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.280 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.280 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.280 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.280 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.280 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s api_c_read_operations 2024-03-26T02:03:49.280 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' api_c_read_operations' 2024-03-26T02:03:49.280 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.280 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.280 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.280 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.280 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.281 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.281 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.281 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.281 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.281 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.281 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.281 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.281 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.281 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.281 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.281 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.281 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.281 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.281 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.281 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.281 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.282 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.282 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.282 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.282 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.282 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.282 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.282 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.282 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.282 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.282 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.282 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.282 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.282 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.282 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.282 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.283 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.283 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.283 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.283 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.283 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.283 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.283 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.283 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.283 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.283 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.283 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.283 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.283 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.283 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.283 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.283 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.284 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.284 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.284 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.284 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.284 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.284 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.284 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.284 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.284 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo api_c_read_operations 2024-03-26T02:03:49.284 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/82923/exe 2024-03-26T02:03:49.284 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.284 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.284 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.284 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.284 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.284 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.285 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.285 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.285 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.285 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.285 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.285 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.285 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.285 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.285 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.285 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.285 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.285 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.285 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.285 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.285 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.285 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.286 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.286 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.286 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.286 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.286 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.286 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.286 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.286 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.286 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.286 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.286 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=api_c_read_operations 2024-03-26T02:03:49.286 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83001 2024-03-26T02:03:49.287 INFO:tasks.workunit.client.0.smithi072.stdout:test api_c_read_operations on pid 83001 2024-03-26T02:03:49.291 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test api_c_read_operations on pid 83001' 2024-03-26T02:03:49.291 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=83001 2024-03-26T02:03:49.291 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.291 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.291 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ api_service: /' 2024-03-26T02:03:49.291 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_api_service.log 2024-03-26T02:03:49.291 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.292 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.292 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.292 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.292 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.292 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.292 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.292 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.292 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.292 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s api_cls_remote_reads 2024-03-26T02:03:49.292 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' api_cls_remote_reads' 2024-03-26T02:03:49.292 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-03-26T02:03:49.292 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.292 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.292 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.292 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.292 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.293 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.293 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.293 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.293 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.293 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.293 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.293 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.293 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.293 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.293 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.293 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.293 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.293 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.293 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.293 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.294 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.294 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.294 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.294 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.294 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-03-26T02:03:49.294 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.294 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.294 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.294 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.294 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.294 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.294 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.294 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.294 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.295 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.295 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.295 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.295 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.295 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.295 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.295 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.295 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.295 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.295 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.295 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.295 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.295 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.295 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/82984/exe 2024-03-26T02:03:49.295 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.296 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.296 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.296 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.296 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.296 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.296 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.296 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.296 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.296 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.296 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.296 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.296 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.296 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.296 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.296 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.296 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.297 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.297 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-03-26T02:03:49.297 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.297 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.297 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-03-26T02:03:49.297 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-03-26T02:03:49.297 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.297 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.297 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.297 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.297 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.297 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.297 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.297 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo api_cls_remote_reads 2024-03-26T02:03:49.297 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.297 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.297 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.298 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.298 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.298 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.298 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.298 INFO:tasks.workunit.client.0.smithi072.stdout:test api_cls_remote_reads on pid 83043 2024-03-26T02:03:49.300 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=api_cls_remote_reads 2024-03-26T02:03:49.301 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83043 2024-03-26T02:03:49.301 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test api_cls_remote_reads on pid 83043' 2024-03-26T02:03:49.301 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=83043 2024-03-26T02:03:49.301 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.301 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.301 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.301 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.301 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.301 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.301 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.301 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.301 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.301 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-03-26T02:03:49.301 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.301 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.302 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.302 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.302 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s list_parallel 2024-03-26T02:03:49.302 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' list_parallel' 2024-03-26T02:03:49.302 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.302 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.302 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.302 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.302 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.302 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.302 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.302 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.302 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.302 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.303 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.303 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.303 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.303 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.303 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.303 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.303 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.303 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.303 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.303 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.303 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.303 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.303 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.303 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.303 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo list_parallel 2024-03-26T02:03:49.304 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.304 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.304 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.304 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.304 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.304 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.304 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.304 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.304 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.304 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.304 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.304 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.304 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.304 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.304 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.304 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.304 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.305 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.305 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.305 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.305 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.305 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=list_parallel 2024-03-26T02:03:49.305 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83076 2024-03-26T02:03:49.305 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test list_parallel on pid 83076' 2024-03-26T02:03:49.305 INFO:tasks.workunit.client.0.smithi072.stdout:test list_parallel on pid 83076 2024-03-26T02:03:49.307 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=83076 2024-03-26T02:03:49.307 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.308 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.308 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-03-26T02:03:49.308 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.308 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.308 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.308 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.308 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.308 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.308 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.308 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.308 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.308 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.308 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.308 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s open_pools_parallel 2024-03-26T02:03:49.308 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.309 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' open_pools_parallel' 2024-03-26T02:03:49.309 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.309 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.309 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.309 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.309 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.309 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.309 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.309 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.309 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.309 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.309 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.309 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.309 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.310 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.310 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.310 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.310 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.310 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.310 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.310 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.310 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.310 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.310 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.310 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.310 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.310 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.310 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.310 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.311 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.311 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.311 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.311 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.311 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.311 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.311 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.311 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.311 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.311 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.311 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.311 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.311 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.311 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.311 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.311 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.311 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.312 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.312 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.312 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.312 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.312 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.312 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.312 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.312 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.312 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.312 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.312 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.312 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.312 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.312 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.312 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.313 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.313 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo open_pools_parallel 2024-03-26T02:03:49.313 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.313 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.313 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.313 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.313 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.313 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.313 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.313 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/82962/exe 2024-03-26T02:03:49.313 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.313 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.313 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.313 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.313 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.314 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.314 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.314 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.314 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.314 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.314 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.314 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.314 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=open_pools_parallel 2024-03-26T02:03:49.314 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-03-26T02:03:49.314 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83107 2024-03-26T02:03:49.314 INFO:tasks.workunit.client.0.smithi072.stdout:test open_pools_parallel on pid 83107 2024-03-26T02:03:49.317 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test open_pools_parallel on pid 83107' 2024-03-26T02:03:49.317 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=83107 2024-03-26T02:03:49.317 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T02:03:49.317 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.318 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-03-26T02:03:49.318 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.318 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.318 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.318 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.318 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.318 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.318 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.318 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.318 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.318 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.318 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.318 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.318 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.319 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.319 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.319 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.319 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.319 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.319 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.319 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.319 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.319 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.319 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.319 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.319 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.319 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.319 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.319 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.320 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.320 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.320 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.320 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.320 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.320 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.320 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.320 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.320 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.320 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-03-26T02:03:49.320 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ api_service_pp: /' 2024-03-26T02:03:49.320 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s delete_pools_parallel 2024-03-26T02:03:49.320 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' delete_pools_parallel' 2024-03-26T02:03:49.320 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.320 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.321 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.321 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.321 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.321 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.321 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.321 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.321 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.321 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.321 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.321 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.321 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.321 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.321 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.321 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.322 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.322 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.322 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.322 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.322 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.322 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.322 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.322 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.322 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.322 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.322 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.322 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.322 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.322 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.323 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.323 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.323 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.323 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.323 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.323 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.323 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.323 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.323 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.323 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.323 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.323 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.323 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.323 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.323 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.324 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.324 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.324 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.324 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.324 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.324 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.324 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.324 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.324 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo delete_pools_parallel 2024-03-26T02:03:49.324 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.324 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.324 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.324 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.324 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.325 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.325 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.325 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.325 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.325 INFO:tasks.workunit.client.0.smithi072.stdout:test delete_pools_parallel on pid 83137 2024-03-26T02:03:49.327 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=delete_pools_parallel 2024-03-26T02:03:49.327 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83137 2024-03-26T02:03:49.327 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test delete_pools_parallel on pid 83137' 2024-03-26T02:03:49.327 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=83137 2024-03-26T02:03:49.327 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.327 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.327 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.327 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T02:03:49.327 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.327 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.327 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.327 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.327 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.328 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.328 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.328 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s cls 2024-03-26T02:03:49.328 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' cls' 2024-03-26T02:03:49.328 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.328 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.328 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.328 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.328 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.328 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.328 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.328 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.328 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.328 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.329 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.329 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.329 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.329 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.329 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.329 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.329 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.329 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.329 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.329 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.329 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.329 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-03-26T02:03:49.329 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.329 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.329 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.329 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.330 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.330 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.330 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.330 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.330 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.330 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.330 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.330 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.330 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.330 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.330 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.330 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.330 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.330 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.331 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.331 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.331 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.331 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo cls 2024-03-26T02:03:49.331 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.331 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.331 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.331 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.331 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.331 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.331 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.331 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.331 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.331 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.331 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.332 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.332 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.332 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.332 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.332 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.332 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.332 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.332 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.332 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.332 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.332 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.332 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.332 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=cls 2024-03-26T02:03:49.333 INFO:tasks.workunit.client.0.smithi072.stdout:test cls on pid 83151 2024-03-26T02:03:49.337 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83151 2024-03-26T02:03:49.337 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test cls on pid 83151' 2024-03-26T02:03:49.337 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=83151 2024-03-26T02:03:49.337 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T02:03:49.337 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.337 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.337 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.337 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.338 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.338 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.338 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/83001/exe 2024-03-26T02:03:49.338 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.338 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.338 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.338 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.338 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.338 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.338 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.338 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.338 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.338 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.338 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.339 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.339 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.339 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.339 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.339 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.339 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.339 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.339 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.339 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.339 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.339 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.339 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.339 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.339 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.339 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.339 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.340 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.340 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.340 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.340 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.340 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.340 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.340 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.340 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.340 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.340 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.340 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.340 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.340 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.340 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.341 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.341 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-03-26T02:03:49.341 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.341 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.341 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.341 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.341 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.341 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.341 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.341 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.341 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.341 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.341 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s cmd 2024-03-26T02:03:49.341 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.341 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.342 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' cmd' 2024-03-26T02:03:49.342 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.342 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.342 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.342 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.342 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.342 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.342 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.342 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.342 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.342 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.342 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.342 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.342 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.342 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.343 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.343 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.343 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-03-26T02:03:49.343 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-03-26T02:03:49.343 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.343 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.343 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.343 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.343 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.343 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.343 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.343 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.343 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.343 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.343 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.344 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-03-26T02:03:49.344 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.344 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.344 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.344 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.344 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.344 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.344 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.344 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.344 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.344 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.344 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.344 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.344 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.344 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.344 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.345 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.345 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.345 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.345 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.345 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.345 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.345 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.345 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.345 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.345 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.345 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.345 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.345 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.345 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.345 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.346 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.346 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.346 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.346 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.346 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.346 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.346 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.346 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.346 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.346 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.346 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.346 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.346 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.346 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.346 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.347 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.347 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.347 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.347 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.347 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo cmd 2024-03-26T02:03:49.347 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.347 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.347 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.347 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.347 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.347 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.347 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.347 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.347 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.347 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.347 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.348 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.348 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.348 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.348 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.348 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.348 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.348 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.348 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.348 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.348 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.348 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.348 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.348 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.348 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.348 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.349 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.349 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.349 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.349 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.349 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.349 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.349 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.349 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.349 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.349 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.349 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.349 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.349 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.349 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.349 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.349 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.350 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.350 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.350 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.350 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.350 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.350 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/83107/exe 2024-03-26T02:03:49.350 INFO:tasks.workunit.client.0.smithi072.stdout:test cmd on pid 83185 2024-03-26T02:03:49.352 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.352 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=cmd 2024-03-26T02:03:49.352 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.352 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.352 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.352 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.352 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.352 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.352 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.352 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.352 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.353 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83185 2024-03-26T02:03:49.353 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.353 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.353 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test cmd on pid 83185' 2024-03-26T02:03:49.353 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.353 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=83185 2024-03-26T02:03:49.353 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.353 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T02:03:49.353 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.353 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.353 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.353 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.353 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.353 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.353 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.354 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.354 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.354 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.354 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.354 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.354 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.354 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.354 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.354 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.354 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.354 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.354 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.354 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.354 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.355 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.355 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.355 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.355 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.355 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.355 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.355 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.355 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.355 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.355 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.355 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.355 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.355 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.355 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.355 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.355 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.356 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.356 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.356 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s handler_error 2024-03-26T02:03:49.356 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-03-26T02:03:49.356 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' handler_error' 2024-03-26T02:03:49.356 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/83043/exe 2024-03-26T02:03:49.356 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.356 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.356 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.356 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.356 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.356 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.356 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.356 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.356 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.356 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.357 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.357 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.357 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.357 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.357 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.357 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.357 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.357 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.357 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.357 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.357 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.357 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-03-26T02:03:49.357 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.357 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.357 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.358 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.358 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.358 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.358 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-03-26T02:03:49.358 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.358 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.358 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.358 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.358 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.358 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.358 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.358 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.358 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.358 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/83076/exe 2024-03-26T02:03:49.358 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo handler_error 2024-03-26T02:03:49.359 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-03-26T02:03:49.359 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-03-26T02:03:49.359 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.359 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=handler_error 2024-03-26T02:03:49.359 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83208 2024-03-26T02:03:49.359 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test handler_error on pid 83208' 2024-03-26T02:03:49.359 INFO:tasks.workunit.client.0.smithi072.stdout:test handler_error on pid 83208 2024-03-26T02:03:49.364 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=83208 2024-03-26T02:03:49.364 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T02:03:49.364 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.364 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.364 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.364 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.365 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.365 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.365 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.365 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.365 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.365 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.365 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.365 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.365 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.365 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.365 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.365 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.365 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.365 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.366 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.366 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-03-26T02:03:49.366 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-03-26T02:03:49.366 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.366 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.366 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.366 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.366 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.366 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.366 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.366 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.366 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.366 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.366 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.366 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.367 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.367 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.367 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.367 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.367 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.367 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.367 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.367 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.367 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.367 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.367 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.367 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.367 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.367 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.367 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.368 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.368 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.368 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.368 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.368 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.368 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.368 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.368 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.368 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.368 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.368 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.368 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.368 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.368 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.369 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.369 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.369 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.369 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.369 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.369 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.369 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.369 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.369 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.369 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.369 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.369 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.369 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.369 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s io 2024-03-26T02:03:49.369 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.370 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.370 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.370 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.370 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.370 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.370 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.370 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' io' 2024-03-26T02:03:49.370 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.370 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.370 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.370 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.370 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.370 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.370 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.371 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.371 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.371 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_list_parallel.log 2024-03-26T02:03:49.371 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.371 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.371 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.371 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.371 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.371 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.371 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ list_parallel: /' 2024-03-26T02:03:49.371 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.371 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.371 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-03-26T02:03:49.371 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.371 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.371 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.372 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.372 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.372 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo io 2024-03-26T02:03:49.372 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.372 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.372 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.372 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.372 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.372 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.372 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.372 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.372 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.372 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.372 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.372 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.373 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.373 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.373 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.373 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.373 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.373 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.373 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.373 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.373 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.373 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.373 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.373 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.373 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.373 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.374 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.374 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.374 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.374 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.374 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.374 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.374 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.374 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.374 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.374 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.374 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.374 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.374 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.374 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.374 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.374 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/83137/exe 2024-03-26T02:03:49.375 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.375 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.375 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-03-26T02:03:49.375 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.375 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.375 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.375 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.375 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.375 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.375 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.375 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.375 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.375 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.375 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.375 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.376 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.376 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.376 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.376 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.376 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.376 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.376 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.376 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.376 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.376 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.376 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.376 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.376 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.376 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.376 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.376 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.376 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.377 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.377 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.377 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.377 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.377 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.377 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.377 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.377 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.377 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.377 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.377 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.377 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-03-26T02:03:49.377 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-03-26T02:03:49.377 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.377 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.377 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.378 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/83151/exe 2024-03-26T02:03:49.378 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.378 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.378 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.378 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.378 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.378 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.378 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.378 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.378 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.378 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.378 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.378 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.378 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.379 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.379 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.379 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-03-26T02:03:49.379 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.379 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.379 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.379 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.379 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.379 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.379 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.379 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.379 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.379 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.379 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.379 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.379 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.380 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.380 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.380 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.380 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.380 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.380 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.380 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.380 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.380 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.380 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.380 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.380 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.380 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.380 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.380 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.380 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.381 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.381 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.381 INFO:tasks.workunit.client.0.smithi072.stdout:test io on pid 83262 2024-03-26T02:03:49.385 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=io 2024-03-26T02:03:49.385 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83262 2024-03-26T02:03:49.385 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test io on pid 83262' 2024-03-26T02:03:49.385 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=83262 2024-03-26T02:03:49.385 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T02:03:49.385 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.386 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.386 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ cls: /' 2024-03-26T02:03:49.386 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_neorados_cls.log 2024-03-26T02:03:49.386 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.386 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.386 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.386 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.386 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.386 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-03-26T02:03:49.386 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s ec_io 2024-03-26T02:03:49.386 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.386 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.386 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.386 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.387 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.387 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.387 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.387 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.387 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.387 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.387 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.387 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.387 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.387 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.387 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.387 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.387 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.387 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.387 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.388 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.388 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.388 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.388 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.388 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.388 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.388 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.388 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.388 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.388 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.388 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/83208/exe 2024-03-26T02:03:49.388 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.388 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.388 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.388 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.389 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.389 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.389 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.389 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.389 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.389 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.389 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.389 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ handler_error: /' 2024-03-26T02:03:49.389 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' ec_io' 2024-03-26T02:03:49.389 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.389 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.389 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.389 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.389 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.389 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.389 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.390 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.390 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.390 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.390 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_neorados_handler_error.log 2024-03-26T02:03:49.390 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.390 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.390 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.390 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.390 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.390 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.390 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.390 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.390 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.390 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.390 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.391 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.391 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.391 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.391 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.391 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.391 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.391 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_neorados_cls 2024-03-26T02:03:49.391 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo ec_io 2024-03-26T02:03:49.391 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.391 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=ec_io 2024-03-26T02:03:49.391 INFO:tasks.workunit.client.0.smithi072.stdout:test ec_io on pid 83321 2024-03-26T02:03:49.396 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83321 2024-03-26T02:03:49.396 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test ec_io on pid 83321' 2024-03-26T02:03:49.396 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=83321 2024-03-26T02:03:49.396 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T02:03:49.396 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.396 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_neorados_handler_error 2024-03-26T02:03:49.396 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.396 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.396 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.397 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.397 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.397 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.397 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.397 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.397 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.397 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.397 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.397 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.397 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.397 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.397 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.397 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.397 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.397 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s list 2024-03-26T02:03:49.398 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' list' 2024-03-26T02:03:49.398 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-03-26T02:03:49.398 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.398 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.398 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.398 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.398 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.398 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.398 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.398 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.398 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.398 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.398 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.398 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.399 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.399 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.399 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.399 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.399 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.399 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.399 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.399 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.399 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.399 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.399 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.399 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.399 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.399 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.399 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.400 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.400 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.400 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.400 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.400 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.400 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.400 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.400 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.400 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.400 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.400 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.400 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.400 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.400 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.400 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.400 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.401 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.401 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.401 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.401 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.401 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.401 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.401 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.401 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.401 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.401 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo list 2024-03-26T02:03:49.401 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.401 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.401 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.401 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.401 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.401 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.402 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.402 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.402 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.402 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.402 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.402 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.402 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.402 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.402 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.402 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.402 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.402 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.402 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.402 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.402 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.403 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.403 INFO:tasks.workunit.client.0.smithi072.stdout:test list on pid 83365 2024-03-26T02:03:49.405 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=list 2024-03-26T02:03:49.405 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83365 2024-03-26T02:03:49.405 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test list on pid 83365' 2024-03-26T02:03:49.405 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=83365 2024-03-26T02:03:49.405 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T02:03:49.405 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.405 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-03-26T02:03:49.405 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.405 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.405 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s ec_list 2024-03-26T02:03:49.405 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.405 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.405 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.405 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.405 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.406 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.406 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.406 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' ec_list' 2024-03-26T02:03:49.406 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.406 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.407 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.407 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.407 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.407 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.407 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.407 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.407 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.407 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.407 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.407 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.407 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.407 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.407 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.407 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.407 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.408 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.408 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.408 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.408 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.408 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.408 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.408 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.408 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.408 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.408 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.408 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.408 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.408 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.408 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.408 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.408 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.409 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.409 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.409 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.409 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.409 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.409 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.409 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.409 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.409 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.409 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.409 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.409 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.409 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo ec_list 2024-03-26T02:03:49.409 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/83185/exe 2024-03-26T02:03:49.409 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.410 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.410 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.410 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.410 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.410 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.410 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.410 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.410 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.410 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.410 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.410 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.410 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.410 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.410 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.410 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.410 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.411 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.411 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.411 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.411 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.411 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.411 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.411 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.411 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.411 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.411 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.411 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.411 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.411 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.411 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.411 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.412 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.412 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.412 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.412 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.412 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.412 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.412 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.412 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.412 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.412 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.412 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.412 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.412 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.412 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.412 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.412 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.412 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.413 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.413 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.413 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.413 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.413 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.413 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.413 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.413 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.413 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.413 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ cmd: /' 2024-03-26T02:03:49.413 INFO:tasks.workunit.client.0.smithi072.stdout:test ec_list on pid 83470 2024-03-26T02:03:49.417 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=ec_list 2024-03-26T02:03:49.417 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83470 2024-03-26T02:03:49.417 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test ec_list on pid 83470' 2024-03-26T02:03:49.417 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=83470 2024-03-26T02:03:49.417 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T02:03:49.417 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.417 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-03-26T02:03:49.418 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s misc 2024-03-26T02:03:49.418 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.418 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' misc' 2024-03-26T02:03:49.418 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo misc 2024-03-26T02:03:49.418 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.418 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.418 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.418 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.418 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.418 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.418 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.418 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.418 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.418 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.419 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.419 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.419 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.419 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.419 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.419 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.419 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.419 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.419 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.419 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.419 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.419 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.419 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.419 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.419 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.419 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.420 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.420 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.420 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.420 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.420 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.420 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.420 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.420 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.420 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.420 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.420 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.420 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.420 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.420 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.420 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.420 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.421 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.421 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.421 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.421 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.421 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.421 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.421 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.421 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.421 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.421 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.421 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.421 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.421 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.421 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.421 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.422 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.422 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.422 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.422 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.422 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.422 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.422 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.422 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.422 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.422 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.422 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.422 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.422 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.422 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_neorados_cmd.log 2024-03-26T02:03:49.422 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/83262/exe 2024-03-26T02:03:49.423 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_neorados_cmd 2024-03-26T02:03:49.423 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=misc 2024-03-26T02:03:49.423 INFO:tasks.workunit.client.0.smithi072.stdout:test misc on pid 83484 2024-03-26T02:03:49.425 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83484 2024-03-26T02:03:49.425 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test misc on pid 83484' 2024-03-26T02:03:49.425 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=83484 2024-03-26T02:03:49.425 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T02:03:49.425 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.425 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-03-26T02:03:49.425 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s pool 2024-03-26T02:03:49.425 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.425 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' pool' 2024-03-26T02:03:49.425 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.425 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.425 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.426 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.426 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.426 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.426 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.426 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.426 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.426 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.426 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.426 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.426 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.426 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.426 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.426 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.426 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.426 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.427 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.427 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.427 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo pool 2024-03-26T02:03:49.427 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.427 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.427 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.427 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.427 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.427 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_neorados_io 2024-03-26T02:03:49.427 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_neorados_io.log 2024-03-26T02:03:49.427 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.427 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ io: /' 2024-03-26T02:03:49.427 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.427 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.427 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.428 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.428 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.428 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.428 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.428 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.428 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.428 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.428 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.428 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.428 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.428 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.428 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.428 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.428 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.428 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.428 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.429 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.429 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.429 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.429 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.429 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.429 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.429 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.429 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.429 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.429 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.429 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.429 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=pool 2024-03-26T02:03:49.429 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.429 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.429 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.430 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.430 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.430 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.430 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.430 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.430 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.430 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.430 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83505 2024-03-26T02:03:49.430 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.430 INFO:tasks.workunit.client.0.smithi072.stdout:test pool on pid 83505 2024-03-26T02:03:49.431 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.431 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test pool on pid 83505' 2024-03-26T02:03:49.432 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.432 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=83505 2024-03-26T02:03:49.432 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.432 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T02:03:49.432 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.432 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.432 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s read_operations 2024-03-26T02:03:49.432 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-03-26T02:03:49.432 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' read_operations' 2024-03-26T02:03:49.432 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.432 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.432 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.432 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.432 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.432 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.433 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.433 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.433 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.433 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.433 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.433 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.433 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.433 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.433 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.433 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.433 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.433 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo read_operations 2024-03-26T02:03:49.433 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.433 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.433 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.434 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.434 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.434 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.434 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.434 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.434 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.434 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.434 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.434 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.434 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.434 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.434 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.434 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.434 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.434 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.435 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.435 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.435 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.435 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.435 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.435 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.435 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.435 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.435 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.435 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.435 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.435 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.435 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.435 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.435 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.435 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.435 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.436 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.436 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.436 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.436 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.436 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.436 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.436 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.436 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.436 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.436 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.436 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.436 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.436 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.436 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.436 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.436 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.437 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.437 INFO:tasks.workunit.client.0.smithi072.stdout:test read_operations on pid 83528 2024-03-26T02:03:49.439 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=read_operations 2024-03-26T02:03:49.439 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83528 2024-03-26T02:03:49.439 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test read_operations on pid 83528' 2024-03-26T02:03:49.439 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=83528 2024-03-26T02:03:49.439 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T02:03:49.439 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.440 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s snapshots 2024-03-26T02:03:49.440 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' snapshots' 2024-03-26T02:03:49.440 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.440 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-03-26T02:03:49.440 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.440 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.440 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.440 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.440 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.440 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.440 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.440 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.440 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.440 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.440 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.441 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.441 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.441 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.441 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.441 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.441 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.441 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.441 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.441 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.441 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.441 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.441 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.441 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.441 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.441 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.441 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.442 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.442 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.442 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.442 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.442 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.442 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.442 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.442 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.442 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.442 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo snapshots 2024-03-26T02:03:49.442 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.442 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.442 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.442 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.442 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.443 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.443 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.443 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.443 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.443 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.443 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.443 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.443 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.443 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.443 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.443 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.443 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.443 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.443 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.443 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.443 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.443 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.444 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.444 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.444 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.444 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.444 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.444 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.444 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.444 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.444 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.444 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.444 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.444 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.444 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.444 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.444 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.444 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.445 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.445 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.445 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.445 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.445 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.445 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.445 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.445 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.445 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.445 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.445 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.445 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.445 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.445 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.445 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.445 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.445 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.446 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.446 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.446 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.446 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.446 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/83365/exe 2024-03-26T02:03:49.446 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.446 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.446 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.446 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.446 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.446 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.446 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.446 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.446 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.447 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.447 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.447 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.447 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/83321/exe 2024-03-26T02:03:49.447 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=snapshots 2024-03-26T02:03:49.447 INFO:tasks.workunit.client.0.smithi072.stdout:test snapshots on pid 83555 2024-03-26T02:03:49.455 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83555 2024-03-26T02:03:49.455 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.455 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test snapshots on pid 83555' 2024-03-26T02:03:49.455 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=83555 2024-03-26T02:03:49.455 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T02:03:49.455 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.455 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s watch_notify 2024-03-26T02:03:49.455 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' watch_notify' 2024-03-26T02:03:49.455 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-03-26T02:03:49.455 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.456 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.456 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.456 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.456 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.456 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.456 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.456 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.456 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.456 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.456 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.456 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.456 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.456 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.456 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.456 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.456 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.457 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.457 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.457 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.457 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.457 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.457 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.457 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.457 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.457 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.457 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.457 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.457 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_neorados_list.log 2024-03-26T02:03:49.457 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_neorados_list 2024-03-26T02:03:49.457 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.457 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.458 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.458 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.458 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.458 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.458 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.458 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.458 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.458 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.458 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.458 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.458 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.458 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.458 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.458 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.459 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.459 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.459 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.459 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.459 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.459 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.459 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.459 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.459 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.459 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.459 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.459 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_neorados_ec_io.log 2024-03-26T02:03:49.459 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.459 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_neorados_ec_io 2024-03-26T02:03:49.459 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.459 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.460 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.460 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.460 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.460 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ list: /' 2024-03-26T02:03:49.460 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ ec_io: /' 2024-03-26T02:03:49.460 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.460 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.460 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.460 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.460 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.460 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.460 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.460 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.460 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.460 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.461 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.461 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.461 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.461 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.461 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.461 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.461 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.461 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.461 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.461 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.461 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.461 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.461 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.461 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.461 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.461 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.462 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.462 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.462 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.462 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.462 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.462 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.462 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.462 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.462 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.462 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.462 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.462 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.462 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.462 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.462 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.463 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.463 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.463 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.463 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.463 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.463 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.463 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.463 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.463 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.463 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.463 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.463 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.463 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.463 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.464 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.464 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.464 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.464 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.464 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.464 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo watch_notify 2024-03-26T02:03:49.464 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.464 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.464 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.464 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.464 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.464 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.464 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.464 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.464 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.464 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.465 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.465 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.465 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.465 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.465 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.465 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.465 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.465 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.465 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.465 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.465 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.465 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.465 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.465 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.465 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.466 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.466 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.466 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.466 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.466 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.466 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.466 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.466 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.466 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.466 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.466 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.466 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.466 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.466 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.466 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.467 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.467 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.467 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.467 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.467 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.467 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.467 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.467 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.467 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.467 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.467 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.467 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.467 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.467 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.467 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.468 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.468 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.468 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.468 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.468 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.468 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.468 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.468 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.468 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.468 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.468 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/83484/exe 2024-03-26T02:03:49.468 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.468 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.468 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.468 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.469 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.469 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.469 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.469 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.469 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.469 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.469 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.469 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.469 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.469 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.469 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.469 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.469 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.469 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.469 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.469 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.470 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/83470/exe 2024-03-26T02:03:49.470 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.470 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.470 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.470 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ misc: /' 2024-03-26T02:03:49.470 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_neorados_misc.log 2024-03-26T02:03:49.470 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.470 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.470 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.470 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.470 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.470 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.470 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.470 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.470 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.471 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.471 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.471 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.471 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.471 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.471 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_neorados_misc 2024-03-26T02:03:49.471 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.471 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.471 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.471 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.471 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.471 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.471 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.472 INFO:tasks.workunit.client.0.smithi072.stdout:test watch_notify on pid 83609 2024-03-26T02:03:49.475 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.475 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.475 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.475 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.475 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.475 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.475 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.475 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.475 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.475 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.475 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.475 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=watch_notify 2024-03-26T02:03:49.476 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83609 2024-03-26T02:03:49.476 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test watch_notify on pid 83609' 2024-03-26T02:03:49.476 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=83609 2024-03-26T02:03:49.476 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T02:03:49.476 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.476 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ ec_list: /' 2024-03-26T02:03:49.476 INFO:tasks.workunit.client.0.smithi072.stderr:++ printf %25s write_operations 2024-03-26T02:03:49.476 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.476 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.476 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.476 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.476 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.476 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.476 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.477 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.477 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.477 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.477 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.477 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.477 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.477 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.477 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.477 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.477 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.477 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.477 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.477 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.477 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.478 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.478 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.478 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.478 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.478 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.478 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.478 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.478 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_neorados_ec_list 2024-03-26T02:03:49.478 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.478 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.478 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.478 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.478 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.478 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.479 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.479 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.479 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.479 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.479 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.479 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.479 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.479 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.479 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.479 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_neorados_ec_list.log 2024-03-26T02:03:49.479 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.479 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.479 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.479 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.479 INFO:tasks.workunit.client.0.smithi072.stderr:+ r=' write_operations' 2024-03-26T02:03:49.480 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.480 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-03-26T02:03:49.480 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/83505/exe 2024-03-26T02:03:49.480 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.480 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.480 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.480 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.480 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.480 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.480 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.480 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.480 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.480 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.480 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.480 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.480 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.481 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.481 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.481 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.481 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.481 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.481 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.481 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.481 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.481 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.481 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.481 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.481 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.481 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.481 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.481 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.482 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.482 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.482 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.482 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.482 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.482 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.482 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.482 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.482 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.482 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.482 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.482 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.482 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.482 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.482 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.483 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.483 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.483 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.483 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.483 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.483 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.483 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.483 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.483 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.483 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.483 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.483 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.483 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.483 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.483 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.483 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.484 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.484 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.484 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.484 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.484 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.484 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.484 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.484 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.484 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.484 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.484 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.484 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.484 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.484 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.484 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.485 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.485 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.485 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.485 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.485 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.485 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.485 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.485 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_neorados_pool.log 2024-03-26T02:03:49.485 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_neorados_pool 2024-03-26T02:03:49.485 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.485 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.485 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.485 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.485 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.485 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ pool: /' 2024-03-26T02:03:49.485 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.485 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/83528/exe 2024-03-26T02:03:49.486 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.486 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.486 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.486 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.486 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.486 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.486 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.486 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.486 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.486 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.486 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.486 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.486 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.486 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.487 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.487 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.487 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.487 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.487 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.487 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.487 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.487 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.487 INFO:tasks.workunit.client.0.smithi072.stderr:++ echo write_operations 2024-03-26T02:03:49.487 INFO:tasks.workunit.client.0.smithi072.stderr:++ awk '{print $1}' 2024-03-26T02:03:49.487 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.487 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.487 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.487 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.487 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.487 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.488 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.488 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.488 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.488 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.488 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.488 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.488 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.488 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.488 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.488 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.488 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.488 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.488 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.488 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.489 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.489 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.489 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.489 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.489 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.489 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.489 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.489 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.489 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.489 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.489 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_neorados_read_operations.log 2024-03-26T02:03:49.489 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ read_operations: /' 2024-03-26T02:03:49.489 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_neorados_read_operations 2024-03-26T02:03:49.490 INFO:tasks.workunit.client.0.smithi072.stdout:test write_operations on pid 83670 2024-03-26T02:03:49.494 INFO:tasks.workunit.client.0.smithi072.stderr:+ ff=write_operations 2024-03-26T02:03:49.494 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83670 2024-03-26T02:03:49.494 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo 'test write_operations on pid 83670' 2024-03-26T02:03:49.494 INFO:tasks.workunit.client.0.smithi072.stderr:+ pids[$f]=83670 2024-03-26T02:03:49.494 INFO:tasks.workunit.client.0.smithi072.stderr:+ ret=0 2024-03-26T02:03:49.495 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T02:03:49.495 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:03:49.495 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82476 2024-03-26T02:03:49.495 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 82476 2024-03-26T02:03:49.495 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/83555/exe 2024-03-26T02:03:49.495 INFO:tasks.workunit.client.0.smithi072.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-03-26T02:03:49.495 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.495 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.495 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.495 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.495 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.495 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.495 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.495 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.496 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.496 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T02:03:49.496 INFO:tasks.workunit.client.0.smithi072.stderr:+ . /etc/bashrc 2024-03-26T02:03:49.496 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -z '' ']' 2024-03-26T02:03:49.496 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.496 INFO:tasks.workunit.client.0.smithi072.stderr:++ BASHRCSOURCED=Y 2024-03-26T02:03:49.496 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.496 INFO:tasks.workunit.client.0.smithi072.stderr:++ shopt -q login_shell 2024-03-26T02:03:49.496 INFO:tasks.workunit.client.0.smithi072.stderr:+++ umask 2024-03-26T02:03:49.496 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T02:03:49.496 INFO:tasks.workunit.client.0.smithi072.stderr:++ SHELL=/bin/bash 2024-03-26T02:03:49.496 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.496 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T02:03:49.496 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.496 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T02:03:49.496 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.497 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.497 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.497 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T02:03:49.497 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.497 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.497 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T02:03:49.497 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.497 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T02:03:49.497 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.497 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.497 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.497 INFO:tasks.workunit.client.0.smithi072.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T02:03:49.497 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.497 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T02:03:49.497 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.498 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.498 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T02:03:49.498 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.498 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.498 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.498 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.498 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_neorados_snapshots 2024-03-26T02:03:49.498 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ snapshots: /' 2024-03-26T02:03:49.498 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_neorados_snapshots.log 2024-03-26T02:03:49.498 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.498 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.498 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.498 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.498 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.498 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.499 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.499 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.499 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.499 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T02:03:49.499 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.499 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.499 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.499 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T02:03:49.499 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T02:03:49.499 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.499 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T02:03:49.499 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.499 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T02:03:49.499 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.499 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.500 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.500 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.500 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T02:03:49.500 INFO:tasks.workunit.client.0.smithi072.stderr:+++ return 2024-03-26T02:03:49.500 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.500 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.500 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T02:03:49.500 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.500 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T02:03:49.500 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export S_COLORS=auto 2024-03-26T02:03:49.500 INFO:tasks.workunit.client.0.smithi072.stderr:+++ S_COLORS=auto 2024-03-26T02:03:49.500 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.500 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T02:03:49.500 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.500 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T02:03:49.501 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.501 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.501 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.501 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.501 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.501 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.501 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.501 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.501 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.501 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.501 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.501 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.501 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.501 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.501 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.501 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.502 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.502 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.502 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.502 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.502 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.502 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.502 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.502 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.502 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.502 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.502 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.502 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.502 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.502 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T02:03:49.502 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T02:03:49.503 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T02:03:49.503 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.503 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T02:03:49.503 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.503 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T02:03:49.503 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T02:03:49.503 INFO:tasks.workunit.client.0.smithi072.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T02:03:49.503 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/83609/exe 2024-03-26T02:03:49.503 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.503 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.503 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.503 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.503 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.503 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.503 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.504 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.504 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.504 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.504 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.504 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T02:03:49.504 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T02:03:49.504 INFO:tasks.workunit.client.0.smithi072.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T02:03:49.504 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.504 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T02:03:49.504 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.504 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T02:03:49.504 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T02:03:49.504 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.504 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T02:03:49.504 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.504 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T02:03:49.505 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.505 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T02:03:49.505 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.505 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T02:03:49.505 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup 2024-03-26T02:03:49.505 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ watch_notify: /' 2024-03-26T02:03:49.505 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-03-26T02:03:49.505 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/locale 2024-03-26T02:03:49.505 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_neorados_watch_notify 2024-03-26T02:03:49.505 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.505 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T02:03:49.505 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.505 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T02:03:49.505 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T02:03:49.506 INFO:tasks.workunit.client.0.smithi072.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T02:03:49.506 INFO:tasks.workunit.client.0.smithi072.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T02:03:49.506 INFO:tasks.workunit.client.0.smithi072.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T02:03:49.506 INFO:tasks.workunit.client.0.smithi072.stderr:++++ LANG=en_US.UTF-8 2024-03-26T02:03:49.506 INFO:tasks.workunit.client.0.smithi072.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T02:03:49.506 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T02:03:49.506 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.506 INFO:tasks.workunit.client.0.smithi072.stderr:+++ LANG=en_US.UTF-8 2024-03-26T02:03:49.506 INFO:tasks.workunit.client.0.smithi072.stderr:+++ unset LANG_backup config 2024-03-26T02:03:49.506 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n '' ']' 2024-03-26T02:03:49.506 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T02:03:49.506 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' dumb = linux ']' 2024-03-26T02:03:49.506 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.506 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T02:03:49.506 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.507 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/less.sh 2024-03-26T02:03:49.507 INFO:tasks.workunit.client.0.smithi072.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T02:03:49.507 INFO:tasks.workunit.client.0.smithi072.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T02:03:49.507 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T02:03:49.507 INFO:tasks.workunit.client.0.smithi072.stderr:++ '[' '' ']' 2024-03-26T02:03:49.507 INFO:tasks.workunit.client.0.smithi072.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T02:03:49.507 INFO:tasks.workunit.client.0.smithi072.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T02:03:49.507 INFO:tasks.workunit.client.0.smithi072.stderr:+++++ readlink /proc/83670/exe 2024-03-26T02:03:49.507 INFO:tasks.workunit.client.0.smithi072.stderr:++++ basename /usr/bin/bash 2024-03-26T02:03:49.507 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_declare='declare -f' 2024-03-26T02:03:49.507 INFO:tasks.workunit.client.0.smithi072.stderr:+++ which_opt=-f 2024-03-26T02:03:49.507 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export which_declare 2024-03-26T02:03:49.507 INFO:tasks.workunit.client.0.smithi072.stderr:+++ export -f which 2024-03-26T02:03:49.507 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset i 2024-03-26T02:03:49.507 INFO:tasks.workunit.client.0.smithi072.stderr:++ unset -f pathmunge 2024-03-26T02:03:49.508 INFO:tasks.workunit.client.0.smithi072.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T02:03:49.508 INFO:tasks.workunit.client.0.smithi072.stderr:+ export PATH 2024-03-26T02:03:49.508 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T02:03:49.508 INFO:tasks.workunit.client.0.smithi072.stderr:+ unset rc 2024-03-26T02:03:49.508 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph_test_neorados_write_operations 2024-03-26T02:03:49.508 INFO:tasks.workunit.client.0.smithi072.stderr:+ sed 's/^/ write_operations: /' 2024-03-26T02:03:49.508 INFO:tasks.workunit.client.0.smithi072.stderr:+ tee ceph_test_neorados_write_operations.log 2024-03-26T02:03:49.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:49 smithi072 ceph-mon[30177]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:49.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:49 smithi072 ceph-mon[41619]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:50.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:49 smithi173 ceph-mon[33839]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:50.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:03:50 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:03:51.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:50 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4017972454' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:03:51.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:50 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2770326193' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-26T02:03:51.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:50 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3366064809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi072-83563-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:03:51.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:03:51.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:50 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2362050383' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi072-82404-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:51.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:50 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3520685527' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi072-82387-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:51.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:50 smithi072 ceph-mon[30177]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T02:03:51.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:50 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4017972454' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:03:51.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:50 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2770326193' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-26T02:03:51.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:50 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3366064809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi072-83563-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:03:51.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:03:51.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:50 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2362050383' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi072-82404-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:51.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:50 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3520685527' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi072-82387-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:51.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:50 smithi072 ceph-mon[41619]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T02:03:51.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:50 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4017972454' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:03:51.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:50 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2770326193' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-26T02:03:51.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:50 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3366064809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi072-83563-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:03:51.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:03:51.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:50 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2362050383' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi072-82404-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:51.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:50 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3520685527' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi072-82387-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:51.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:50 smithi173 ceph-mon[33839]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T02:03:54.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:54 smithi072 ceph-mon[30177]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:54.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:54 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi072-83605-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:03:54.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:54 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:03:54.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:54 smithi072 ceph-mon[41619]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:54.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:54 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi072-83605-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:03:54.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:54 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:03:54.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:54 smithi173 ceph-mon[33839]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:03:54.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:54 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi072-83605-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:03:54.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:54 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:03:55.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:03:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:03:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:03:57.949 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-03-26T02:03:57.949 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [----------] Global test environment set-up. 2024-03-26T02:03:57.949 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [----------] 12 tests from AsioRados 2024-03-26T02:03:57.949 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-03-26T02:03:57.949 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (2 ms) 2024-03-26T02:03:57.949 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-03-26T02:03:57.949 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (0 ms) 2024-03-26T02:03:57.949 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-03-26T02:03:57.949 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (1 ms) 2024-03-26T02:03:57.950 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-03-26T02:03:57.950 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (4 ms) 2024-03-26T02:03:57.950 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-03-26T02:03:57.950 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (4 ms) 2024-03-26T02:03:57.950 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-03-26T02:03:57.950 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (2 ms) 2024-03-26T02:03:57.950 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-03-26T02:03:57.950 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (0 ms) 2024-03-26T02:03:57.950 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-03-26T02:03:57.950 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (1 ms) 2024-03-26T02:03:57.950 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-03-26T02:03:57.950 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (1 ms) 2024-03-26T02:03:57.950 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-03-26T02:03:57.950 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (1 ms) 2024-03-26T02:03:57.951 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-03-26T02:03:57.951 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (1 ms) 2024-03-26T02:03:57.951 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-03-26T02:03:57.951 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (1 ms) 2024-03-26T02:03:57.951 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [----------] 12 tests from AsioRados (18 ms total) 2024-03-26T02:03:57.951 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: 2024-03-26T02:03:57.951 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [----------] Global test environment tear-down 2024-03-26T02:03:57.951 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (8850 ms total) 2024-03-26T02:03:57.951 INFO:tasks.workunit.client.0.smithi072.stdout: api_asio: [ PASSED ] 12 tests. 2024-03-26T02:03:58.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:57 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:03:58.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:57 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/896664651' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi072-82496-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:57 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4197596331' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi072-82429-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:57 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi072-82492-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:57 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/963030147' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi072-82437-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:57 smithi173 ceph-mon[33839]: pgmap v66: 356 pgs: 160 unknown, 64 creating+peering, 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:57 smithi173 ceph-mon[33839]: pgmap v67: 356 pgs: 160 unknown, 64 creating+peering, 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:03:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:57 smithi173 ceph-mon[33839]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T02:03:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:57 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4017972454' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:03:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:57 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi072-83563-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:03:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:57 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi072-82496-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:57 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi072-82429-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:57 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi072-82492-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:57 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi072-82404-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:57 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi072-82387-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:57 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi072-82437-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:57 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:03:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:57 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi072-83605-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:03:58.156 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-03-26T02:03:58.156 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [----------] Global test environment set-up. 2024-03-26T02:03:58.156 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [----------] 7 tests from LibRadosList 2024-03-26T02:03:58.157 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-03-26T02:03:58.157 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [ OK ] LibRadosList.ListObjects (130 ms) 2024-03-26T02:03:58.157 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-03-26T02:03:58.157 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (20 ms) 2024-03-26T02:03:58.157 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-03-26T02:03:58.157 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: myset foo1,foo2,foo3 2024-03-26T02:03:58.157 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: foo1 2024-03-26T02:03:58.157 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: foo2 2024-03-26T02:03:58.157 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: foo3 2024-03-26T02:03:58.157 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: myset foo1,foo4,foo5 2024-03-26T02:03:58.157 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: foo4 2024-03-26T02:03:58.158 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: foo5 2024-03-26T02:03:58.158 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: foo1 2024-03-26T02:03:58.158 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: myset foo6,foo7 2024-03-26T02:03:58.158 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: foo7 2024-03-26T02:03:58.158 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: foo6 2024-03-26T02:03:58.158 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-26T02:03:58.158 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: ns1:foo4 2024-03-26T02:03:58.158 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: ns1:foo5 2024-03-26T02:03:58.158 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: ns2:foo7 2024-03-26T02:03:58.159 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: ns2:foo6 2024-03-26T02:03:58.159 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: ns1:foo1 2024-03-26T02:03:58.159 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: :foo1 2024-03-26T02:03:58.159 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: :foo2 2024-03-26T02:03:58.159 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: :foo3 2024-03-26T02:03:58.159 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (57 ms) 2024-03-26T02:03:58.159 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-03-26T02:03:58.159 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 1 0 2024-03-26T02:03:58.159 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 10 0 2024-03-26T02:03:58.159 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 13 0 2024-03-26T02:03:58.159 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 7 0 2024-03-26T02:03:58.159 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 14 0 2024-03-26T02:03:58.159 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 0 0 2024-03-26T02:03:58.160 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 15 0 2024-03-26T02:03:58.160 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 11 0 2024-03-26T02:03:58.160 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 5 0 2024-03-26T02:03:58.160 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 8 0 2024-03-26T02:03:58.160 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 6 0 2024-03-26T02:03:58.160 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 3 0 2024-03-26T02:03:58.160 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 4 0 2024-03-26T02:03:58.160 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 12 0 2024-03-26T02:03:58.160 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 9 0 2024-03-26T02:03:58.160 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 2 0 2024-03-26T02:03:58.160 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-26T02:03:58.160 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (53 ms) 2024-03-26T02:03:58.160 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-03-26T02:03:58.161 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: x cursor=MIN 2024-03-26T02:03:58.161 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > oid=1 cursor=13:02547ec2:::1:head 2024-03-26T02:03:58.161 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > oid=10 cursor=13:52ea6a34:::10:head 2024-03-26T02:03:58.161 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > oid=13 cursor=13:566253c9:::13:head 2024-03-26T02:03:58.161 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > oid=7 cursor=13:5c6b0b28:::7:head 2024-03-26T02:03:58.161 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > oid=14 cursor=13:62a1935d:::14:head 2024-03-26T02:03:58.161 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > oid=0 cursor=13:6cac518f:::0:head 2024-03-26T02:03:58.161 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > oid=15 cursor=13:863748b0:::15:head 2024-03-26T02:03:58.161 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > oid=11 cursor=13:89d3ae78:::11:head 2024-03-26T02:03:58.161 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > oid=5 cursor=13:b29083e3:::5:head 2024-03-26T02:03:58.161 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > oid=8 cursor=13:bd63b0f1:::8:head 2024-03-26T02:03:58.161 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > oid=6 cursor=13:c4fdafeb:::6:head 2024-03-26T02:03:58.162 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > oid=3 cursor=13:cfc208b3:::3:head 2024-03-26T02:03:58.162 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > oid=4 cursor=13:d83876eb:::4:head 2024-03-26T02:03:58.162 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > oid=12 cursor=13:de5d7c5f:::12:head 2024-03-26T02:03:58.162 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > oid=9 cursor=13:e960b815:::9:head 2024-03-26T02:03:58.162 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > oid=2 cursor=13:f905c69b:::2:head 2024-03-26T02:03:58.162 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: FIRST> seek to MIN oid=1 2024-03-26T02:03:58.162 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : oid=1 cursor=13:02547ec2:::1:head 2024-03-26T02:03:58.162 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:02547ec2:::1:head 2024-03-26T02:03:58.162 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:02547ec2:::1:head -> 1 2024-03-26T02:03:58.162 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : oid=10 cursor=13:52ea6a34:::10:head 2024-03-26T02:03:58.162 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:52ea6a34:::10:head 2024-03-26T02:03:58.162 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:52ea6a34:::10:head -> 10 2024-03-26T02:03:58.163 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : oid=13 cursor=13:566253c9:::13:head 2024-03-26T02:03:58.163 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:566253c9:::13:head 2024-03-26T02:03:58.163 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:566253c9:::13:head -> 13 2024-03-26T02:03:58.163 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : oid=7 cursor=13:5c6b0b28:::7:head 2024-03-26T02:03:58.163 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:5c6b0b28:::7:head 2024-03-26T02:03:58.163 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:5c6b0b28:::7:head -> 7 2024-03-26T02:03:58.211 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : oid=14 cursor=13:62a1935d:::14:head 2024-03-26T02:03:58.212 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:62a1935d:::14:head 2024-03-26T02:03:58.212 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:62a1935d:::14:head -> 14 2024-03-26T02:03:58.212 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : oid=0 cursor=13:6cac518f:::0:head 2024-03-26T02:03:58.212 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:6cac518f:::0:head 2024-03-26T02:03:58.212 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:6cac518f:::0:head -> 0 2024-03-26T02:03:58.212 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : oid=15 cursor=13:863748b0:::15:head 2024-03-26T02:03:58.212 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:863748b0:::15:head 2024-03-26T02:03:58.212 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:863748b0:::15:head -> 15 2024-03-26T02:03:58.212 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : oid=11 cursor=13:89d3ae78:::11:head 2024-03-26T02:03:58.212 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:89d3ae78:::11:head 2024-03-26T02:03:58.212 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:89d3ae78:::11:head -> 11 2024-03-26T02:03:58.212 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : oid=5 cursor=13:b29083e3:::5:head 2024-03-26T02:03:58.212 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:b29083e3:::5:head 2024-03-26T02:03:58.212 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:b29083e3:::5:head -> 5 2024-03-26T02:03:58.213 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : oid=8 cursor=13:bd63b0f1:::8:head 2024-03-26T02:03:58.213 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:bd63b0f1:::8:head 2024-03-26T02:03:58.213 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:bd63b0f1:::8:head -> 8 2024-03-26T02:03:58.213 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : oid=6 cursor=13:c4fdafeb:::6:head 2024-03-26T02:03:58.213 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:c4fdafeb:::6:head 2024-03-26T02:03:58.213 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:c4fdafeb:::6:head -> 6 2024-03-26T02:03:58.213 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : oid=3 cursor=13:cfc208b3:::3:head 2024-03-26T02:03:58.213 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:cfc208b3:::3:head 2024-03-26T02:03:58.213 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:cfc208b3:::3:head -> 3 2024-03-26T02:03:58.213 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : oid=4 cursor=13:d83876eb:::4:head 2024-03-26T02:03:58.213 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:d83876eb:::4:head 2024-03-26T02:03:58.213 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:d83876eb:::4:head -> 4 2024-03-26T02:03:58.213 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : oid=12 cursor=13:de5d7c5f:::12:head 2024-03-26T02:03:58.213 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:de5d7c5f:::12:head 2024-03-26T02:03:58.214 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:de5d7c5f:::12:head -> 12 2024-03-26T02:03:58.214 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : oid=9 cursor=13:e960b815:::9:head 2024-03-26T02:03:58.214 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:e960b815:::9:head 2024-03-26T02:03:58.214 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:e960b815:::9:head -> 9 2024-03-26T02:03:58.214 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : oid=2 cursor=13:f905c69b:::2:head 2024-03-26T02:03:58.214 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:f905c69b:::2:head 2024-03-26T02:03:58.214 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:f905c69b:::2:head -> 2 2024-03-26T02:03:58.214 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:02547ec2:::1:head 2024-03-26T02:03:58.214 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : cursor()=13:02547ec2:::1:head expected=13:02547ec2:::1:head 2024-03-26T02:03:58.214 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:02547ec2:::1:head -> 1 2024-03-26T02:03:58.214 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : entry=1 expected=1 2024-03-26T02:03:58.214 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:c4fdafeb:::6:head 2024-03-26T02:03:58.214 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : cursor()=13:c4fdafeb:::6:head expected=13:c4fdafeb:::6:head 2024-03-26T02:03:58.214 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:c4fdafeb:::6:head -> 6 2024-03-26T02:03:58.215 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : entry=6 expected=6 2024-03-26T02:03:58.215 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:bd63b0f1:::8:head 2024-03-26T02:03:58.215 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : cursor()=13:bd63b0f1:::8:head expected=13:bd63b0f1:::8:head 2024-03-26T02:03:58.215 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:bd63b0f1:::8:head -> 8 2024-03-26T02:03:58.215 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : entry=8 expected=8 2024-03-26T02:03:58.215 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:de5d7c5f:::12:head 2024-03-26T02:03:58.215 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : cursor()=13:de5d7c5f:::12:head expected=13:de5d7c5f:::12:head 2024-03-26T02:03:58.215 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:de5d7c5f:::12:head -> 12 2024-03-26T02:03:58.215 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : entry=12 expected=12 2024-03-26T02:03:58.215 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:d83876eb:::4:head 2024-03-26T02:03:58.215 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : cursor()=13:d83876eb:::4:head expected=13:d83876eb:::4:head 2024-03-26T02:03:58.215 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:d83876eb:::4:head -> 4 2024-03-26T02:03:58.215 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : entry=4 expected=4 2024-03-26T02:03:58.215 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:cfc208b3:::3:head 2024-03-26T02:03:58.215 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : cursor()=13:cfc208b3:::3:head expected=13:cfc208b3:::3:head 2024-03-26T02:03:58.216 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:cfc208b3:::3:head -> 3 2024-03-26T02:03:58.216 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : entry=3 expected=3 2024-03-26T02:03:58.216 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:b29083e3:::5:head 2024-03-26T02:03:58.216 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : cursor()=13:b29083e3:::5:head expected=13:b29083e3:::5:head 2024-03-26T02:03:58.216 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:b29083e3:::5:head -> 5 2024-03-26T02:03:58.216 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : entry=5 expected=5 2024-03-26T02:03:58.216 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:89d3ae78:::11:head 2024-03-26T02:03:58.216 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : cursor()=13:89d3ae78:::11:head expected=13:89d3ae78:::11:head 2024-03-26T02:03:58.216 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:89d3ae78:::11:head -> 11 2024-03-26T02:03:58.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:03:58.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/896664651' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi072-82496-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4197596331' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi072-82429-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi072-82492-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/963030147' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi072-82437-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[30177]: pgmap v66: 356 pgs: 160 unknown, 64 creating+peering, 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:58.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[30177]: pgmap v67: 356 pgs: 160 unknown, 64 creating+peering, 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:03:58.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[30177]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T02:03:58.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4017972454' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:03:58.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi072-83563-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:03:58.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi072-82496-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi072-82429-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi072-82492-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi072-82404-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi072-82387-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi072-82437-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:03:58.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi072-83605-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:03:58.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:03:58.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/896664651' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi072-82496-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4197596331' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi072-82429-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi072-82492-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/963030147' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi072-82437-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[41619]: pgmap v66: 356 pgs: 160 unknown, 64 creating+peering, 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:03:58.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[41619]: pgmap v67: 356 pgs: 160 unknown, 64 creating+peering, 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:03:58.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[41619]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T02:03:58.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4017972454' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:03:58.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi072-83563-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:03:58.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi072-82496-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi072-82429-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi072-82492-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi072-82404-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi072-82387-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi072-82437-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:58.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:03:58.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:57 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi072-83605-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:03:59.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3463021879' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi072-82692-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3860683801' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi072-82604-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1444380195' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2300601815' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi072-82904-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3960797341' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi072-82723-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1991867382' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi072-82728-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4058785840' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi072-82705-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1194997556' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83203-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/504701236' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi072-82877-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: pgmap v69: 836 pgs: 199 unknown, 474 creating+peering, 163 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T02:03:59.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: pgmap v70: 836 pgs: 199 unknown, 474 creating+peering, 163 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T02:03:59.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T02:03:59.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: Health check failed: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:03:59.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4017972454' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:03:59.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi072-83605-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi072-83605-1"}]: dispatch 2024-03-26T02:03:59.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi072-82692-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2875234398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi072-82604-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi072-82705-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83203-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi072-82904-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi072-82877-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1232774508' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82958-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi072-82723-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi072-82728-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi072-83563-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:03:59.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi072-82496-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:03:59.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi072-82429-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:03:59.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi072-82492-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:03:59.067 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi072-82404-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:03:59.067 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi072-82387-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:03:59.067 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi072-82437-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:03:59.067 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3366064809' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi072-83563-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi072-83563-1"}]: dispatch 2024-03-26T02:03:59.067 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T02:03:59.067 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2805486639' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi072-83158-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.067 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2178648948' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82972-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.067 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:03:59.067 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi072-83563-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi072-83563-1"}]: dispatch 2024-03-26T02:03:59.067 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82972-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.067 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2804377020' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T02:03:59.067 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi072-83158-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.067 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:03:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:03:59.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3463021879' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi072-82692-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3860683801' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi072-82604-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1444380195' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2300601815' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi072-82904-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3960797341' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi072-82723-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1991867382' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi072-82728-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4058785840' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi072-82705-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1194997556' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83203-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/504701236' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi072-82877-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: pgmap v69: 836 pgs: 199 unknown, 474 creating+peering, 163 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T02:03:59.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: pgmap v70: 836 pgs: 199 unknown, 474 creating+peering, 163 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T02:03:59.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T02:03:59.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: Health check failed: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:03:59.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4017972454' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:03:59.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi072-83605-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi072-83605-1"}]: dispatch 2024-03-26T02:03:59.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi072-82692-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3463021879' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi072-82692-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3860683801' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi072-82604-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1444380195' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2300601815' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi072-82904-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3960797341' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi072-82723-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1991867382' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi072-82728-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4058785840' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi072-82705-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1194997556' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83203-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/504701236' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi072-82877-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: pgmap v69: 836 pgs: 199 unknown, 474 creating+peering, 163 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T02:03:59.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: pgmap v70: 836 pgs: 199 unknown, 474 creating+peering, 163 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T02:03:59.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T02:03:59.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: Health check failed: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:03:59.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4017972454' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:03:59.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi072-83605-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi072-83605-1"}]: dispatch 2024-03-26T02:03:59.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi072-82692-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2875234398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi072-82604-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi072-82705-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83203-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi072-82904-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi072-82877-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1232774508' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82958-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi072-82723-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi072-82728-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi072-83563-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:03:59.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi072-82496-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:03:59.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi072-82429-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:03:59.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi072-82492-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:03:59.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi072-82404-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:03:59.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi072-82387-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:03:59.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi072-82437-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:03:59.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3366064809' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi072-83563-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi072-83563-1"}]: dispatch 2024-03-26T02:03:59.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T02:03:59.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2805486639' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi072-83158-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2178648948' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82972-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:03:59.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi072-83563-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi072-83563-1"}]: dispatch 2024-03-26T02:03:59.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82972-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2804377020' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T02:03:59.225 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi072-83158-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.225 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:03:59.225 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.225 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2875234398' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.225 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi072-82604-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.225 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi072-82705-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.225 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.225 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83203-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.225 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi072-82904-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.225 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi072-82877-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.226 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1232774508' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82958-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.226 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi072-82723-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.226 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi072-82728-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.226 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi072-83563-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:03:59.226 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi072-82496-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:03:59.226 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi072-82429-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:03:59.226 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi072-82492-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:03:59.226 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi072-82404-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:03:59.226 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi072-82387-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:03:59.226 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi072-82437-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:03:59.226 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3366064809' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi072-83563-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi072-83563-1"}]: dispatch 2024-03-26T02:03:59.226 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T02:03:59.227 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2805486639' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi072-83158-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.227 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2178648948' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82972-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.227 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:03:59.227 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi072-83563-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi072-83563-1"}]: dispatch 2024-03-26T02:03:59.227 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82972-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.227 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2804377020' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T02:03:59.227 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi072-83158-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:03:59.227 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:03:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:03:59.906 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : entry=11 e list_parallel: process_1_[83311]: starting. 2024-03-26T02:03:59.907 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_1_[83311]: creating pool ceph_test_rados_list_parallel.smithi072-83217 2024-03-26T02:03:59.907 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_1_[83311]: created object 0... 2024-03-26T02:03:59.907 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_1_[83311]: created object 25... 2024-03-26T02:03:59.907 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_1_[83311]: created object 49... 2024-03-26T02:03:59.907 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_1_[83311]: finishing. 2024-03-26T02:03:59.907 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_1_[83311]: shutting down. 2024-03-26T02:03:59.907 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_2_[83312]: starting. 2024-03-26T02:03:59.907 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_2_[83312]: listing objects. 2024-03-26T02:03:59.907 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_2_[83312]: listed object 0... 2024-03-26T02:03:59.907 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_2_[83312]: listed object 25... 2024-03-26T02:03:59.907 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_2_[83312]: saw 50 objects 2024-03-26T02:03:59.908 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_2_[83312]: shutting down. 2024-03-26T02:03:59.908 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:03:59.908 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_3_[84329]: starting. 2024-03-26T02:03:59.908 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_3_[84329]: creating pool ceph_test_rados_list_parallel.smithi072-83217 2024-03-26T02:03:59.908 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_3_[84329]: created object 0... 2024-03-26T02:03:59.908 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_3_[84329]: created object 25... 2024-03-26T02:03:59.908 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_3_[84329]: created object 49... 2024-03-26T02:03:59.908 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_3_[84329]: finishing. 2024-03-26T02:03:59.908 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_3_[84329]: shutting down. 2024-03-26T02:03:59.908 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:03:59.908 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_4_[84330]: starting. 2024-03-26T02:03:59.908 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_4_[84330]: listing objects. 2024-03-26T02:03:59.908 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_4_[84330]: listed object 0... 2024-03-26T02:03:59.908 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_4_[84330]: listed object 25... 2024-03-26T02:03:59.908 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_4_[84330]: saw 43 objects 2024-03-26T02:03:59.909 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_4_[84330]: shutting down. 2024-03-26T02:03:59.909 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:03:59.909 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_5_[84331]: starting. 2024-03-26T02:03:59.909 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_5_[84331]: removed 25 objects... 2024-03-26T02:03:59.909 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_5_[84331]: removed half of the objects 2024-03-26T02:03:59.909 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_5_[84331]: removed 50 objects... 2024-03-26T02:03:59.909 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_5_[84331]: removed 50 objects 2024-03-26T02:03:59.909 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_5_[84331]: shutting down. 2024-03-26T02:03:59.909 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:03:59.909 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:03:59.909 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_6_[84478]: starting. 2024-03-26T02:03:59.909 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_6_[84478]: creating pool ceph_test_rados_list_parallel.smithi072-83217 2024-03-26T02:03:59.909 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_6_[84478]: created object 0... 2024-03-26T02:03:59.909 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_6_[84478]: created object 25... 2024-03-26T02:03:59.909 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_6_[84478]: created object 49... 2024-03-26T02:03:59.910 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_6_[84478]: finishing. 2024-03-26T02:03:59.910 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_6_[84478]: shutting down. 2024-03-26T02:03:59.910 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:03:59.910 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:03:59.910 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_7_[84479]: starting. 2024-03-26T02:03:59.910 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_7_[84479]: listing objects. 2024-03-26T02:03:59.910 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_7_[84479]: listed object 0... 2024-03-26T02:03:59.910 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_7_[84479]: listed object 25... 2024-03-26T02:03:59.910 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_7_[84479]: listed object 50... 2024-03-26T02:03:59.910 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_7_[84479]: saw 58 objects 2024-03-26T02:03:59.910 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_7_[84479]: shutting down. 2024-03-26T02:03:59.910 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:03:59.910 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:03:59.910 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_8_[84480]: starting. 2024-03-26T02:03:59.910 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_8_[84480]: added 25 objects... 2024-03-26T02:03:59.911 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_8_[84480]: added half of the objects 2024-03-26T02:03:59.911 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_8_[84480]: added 50 objects... 2024-03-26T02:03:59.911 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_8_[84480]: added 50 objects 2024-03-26T02:03:59.911 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_8_[84480]: shutting down. 2024-03-26T02:03:59.911 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:03:59.911 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:00.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:00 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi072-82692-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:00 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2875234398' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:00 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi072-82604-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:00 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi072-82705-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:00 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:00 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83203-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:00 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi072-82904-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:00 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi072-82877-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1232774508' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82958-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:00 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi072-82723-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:00 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi072-82728-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:00 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:00.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:00 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82972-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:00 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi072-83158-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4017972454' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi072-82938-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:04:00.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:00.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:00 smithi173 ceph-mon[33839]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T02:04:00.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:00 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi072-82938-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:04:00.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4133273429' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:00.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/711058968' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:00.350 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ********************** api_cmd_pp: Running main() from gmock_main.cc 2024-03-26T02:04:00.350 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-03-26T02:04:00.350 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-03-26T02:04:00.350 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-03-26T02:04:00.350 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-03-26T02:04:00.351 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (1641 ms) 2024-03-26T02:04:00.351 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-03-26T02:04:00.351 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (25 ms) 2024-03-26T02:04:00.351 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-03-26T02:04:00.351 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (9448 ms) 2024-03-26T02:04:00.351 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (11114 ms total) 2024-03-26T02:04:00.351 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd_pp: 2024-03-26T02:04:00.351 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-03-26T02:04:00.351 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (11114 ms total) 2024-03-26T02:04:00.351 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-03-26T02:04:00.355 INFO:tasks.workunit.client.0.smithi072.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-03-26T02:04:00.355 INFO:tasks.workunit.client.0.smithi072.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-03-26T02:04:00.355 INFO:tasks.workunit.client.0.smithi072.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-03-26T02:04:00.355 INFO:tasks.workunit.client.0.smithi072.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-03-26T02:04:00.355 INFO:tasks.workunit.client.0.smithi072.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (11017 ms) 2024-03-26T02:04:00.355 INFO:tasks.workunit.client.0.smithi072.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (11017 ms total) 2024-03-26T02:04:00.356 INFO:tasks.workunit.client.0.smithi072.stdout: api_cls_remote_reads: 2024-03-26T02:04:00.356 INFO:tasks.workunit.client.0.smithi072.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-03-26T02:04:00.356 INFO:tasks.workunit.client.0.smithi072.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (11017 ms total) 2024-03-26T02:04:00.356 INFO:tasks.workunit.client.0.smithi072.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-03-26T02:04:00.416 INFO:tasks.workunit.client.0.smithi072.stdout:********* 2024-03-26T02:04:00.416 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_9_[84528]: starting. 2024-03-26T02:04:00.416 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_9_[84528]: creating pool ceph_test_rados_list_parallel.smithi072-83217 2024-03-26T02:04:00.416 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_9_[84528]: created object 0... 2024-03-26T02:04:00.416 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_9_[84528]: created object 25... 2024-03-26T02:04:00.416 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_9_[84528]: created object 49... 2024-03-26T02:04:00.416 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_9_[84528]: finishing. 2024-03-26T02:04:00.416 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_9_[84528]: shutting down. 2024-03-26T02:04:00.416 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:00.416 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:00.416 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:00.416 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_13_[84532]: starting. 2024-03-26T02:04:00.417 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_13_[84532]: removed 25 objects... 2024-03-26T02:04:00.417 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_13_[84532]: removed half of the objects 2024-03-26T02:04:00.417 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_13_[84532]: removed 50 objects... 2024-03-26T02:04:00.417 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_13_[84532]: removed 50 objects 2024-03-26T02:04:00.417 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_13_[84532]: shutting down. 2024-03-26T02:04:00.417 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:00.417 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:00.417 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:00.417 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_12_[84531]: starting. 2024-03-26T02:04:00.417 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_12_[84531]: added 25 objects... 2024-03-26T02:04:00.417 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_12_[84531]: added half of the objects 2024-03-26T02:04:00.417 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_12_[84531]: added 50 objects... 2024-03-26T02:04:00.417 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_12_[84531]: added 50 objects 2024-03-26T02:04:00.417 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_12_[84531]: shutting down. 2024-03-26T02:04:00.417 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:00.418 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:00.418 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:00.418 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_10_[84529]: starting. 2024-03-26T02:04:00.418 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_10_[84529]: listing objects. 2024-03-26T02:04:00.418 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_10_[84529]: listed object 0... 2024-03-26T02:04:00.418 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_10_[84529]: listed object 25... 2024-03-26T02:04:00.418 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_10_[84529]: listed object 50... 2024-03-26T02:04:00.418 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_10_[84529]: listed object 75... 2024-03-26T02:04:00.418 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_10_[84529]: saw 100 objects 2024-03-26T02:04:00.418 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_10_[84529]: shutting down. 2024-03-26T02:04:00.418 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:00.418 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:00.418 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:00.419 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_11_[84530]: starting. 2024-03-26T02:04:00.419 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_11_[84530]: added 25 objects... 2024-03-26T02:04:00.419 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_11_[84530]: added half of the objects 2024-03-26T02:04:00.419 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_11_[84530]: added 50 objects... 2024-03-26T02:04:00.419 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_11_[84530]: added 50 objects 2024-03-26T02:04:00.419 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_11_[84530]: shutting down. 2024-03-26T02:04:00.419 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:00.419 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:00.419 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:00.419 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:00.419 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_14_[84627]: starting. 2024-03-26T02:04:00.419 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_14_[84627]: creating pool ceph_test_rados_list_parallel.smithi072-83217 2024-03-26T02:04:00.419 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_14_[84627]: created object 0... 2024-03-26T02:04:00.419 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_14_[84627]: created object 25... 2024-03-26T02:04:00.419 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_14_[84627]: created object 49... 2024-03-26T02:04:00.419 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_14_[84627]: finishing. 2024-03-26T02:04:00.420 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_14_[84627]: shutting down. 2024-03-26T02:04:00.420 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:00.420 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:00.420 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:00.420 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:00.420 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_15_[84628]: starting. 2024-03-26T02:04:00.420 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_15_[84628]: listing objects. 2024-03-26T02:04:00.420 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_15_[84628]: listed object 0... 2024-03-26T02:04:00.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi072-82692-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2875234398' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi072-82604-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi072-82705-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83203-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi072-82904-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi072-82877-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1232774508' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82958-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi072-82723-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi072-82728-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:00.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82972-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi072-83158-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4017972454' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi072-82938-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:04:00.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:00.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[30177]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T02:04:00.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi072-82692-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2875234398' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi072-82604-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi072-82705-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83203-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi072-82904-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi072-82877-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1232774508' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82958-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi072-82723-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi072-82728-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:00.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82972-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi072-83158-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4017972454' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi072-82938-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:04:00.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:00.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[41619]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T02:04:00.473 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi072-82938-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:04:00.473 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4133273429' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:00.473 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/711058968' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:00.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi072-82938-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:04:00.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4133273429' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:00.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/711058968' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:00.647 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_15_[84628]: listed object api_cmd: Running main() from gmock_main.cc 2024-03-26T02:04:00.647 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-03-26T02:04:00.647 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: [----------] Global test environment set-up. 2024-03-26T02:04:00.647 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-03-26T02:04:00.647 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-03-26T02:04:00.647 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (52 ms) 2024-03-26T02:04:00.647 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-03-26T02:04:00.647 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (59 ms) 2024-03-26T02:04:00.648 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-03-26T02:04:00.648 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (11281 ms) 2024-03-26T02:04:00.648 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-03-26T02:04:00.648 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:03:58.820230+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi072-82692-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.648 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:03:58.820312+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.648 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:03:58.820405+0000 mon.a [INF] from='client.? 172.21.15.72:0/2875234398' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.648 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:03:58.820485+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi072-82604-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.648 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:03:58.820576+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi072-82705-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.648 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:03:58.820660+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.648 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:03:58.820751+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83203-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.648 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:03:58.820844+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi072-82904-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.648 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:03:58.820919+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi072-82877-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.649 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:03:58.820993+0000 mon.a [INF] from='client.? 172.21.15.72:0/1232774508' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82958-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.649 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:03:58.821055+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi072-82723-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.649 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:03:58.821105+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi072-82728-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:00.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:04:00 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:04:01.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:01 smithi072 ceph-mon[30177]: pgmap v73: 1092 pgs: 583 unknown, 377 creating+peering, 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:04:01.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4131072319' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T02:04:01.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi072-83605-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi072-83605-1"}]': finished 2024-03-26T02:04:01.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:01 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi072-83563-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi072-83563-1"}]': finished 2024-03-26T02:04:01.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:01.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/692566042' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:01.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:01 smithi072 ceph-mon[30177]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T02:04:01.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4045723459' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T02:04:01.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:01 smithi072 ceph-mon[41619]: pgmap v73: 1092 pgs: 583 unknown, 377 creating+peering, 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:04:01.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4131072319' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T02:04:01.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi072-83605-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi072-83605-1"}]': finished 2024-03-26T02:04:01.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:01 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi072-83563-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi072-83563-1"}]': finished 2024-03-26T02:04:01.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:01.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/692566042' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:01.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:01 smithi072 ceph-mon[41619]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T02:04:01.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4045723459' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T02:04:01.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:01 smithi173 ceph-mon[33839]: pgmap v73: 1092 pgs: 583 unknown, 377 creating+peering, 132 active+clean; 582 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:04:01.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4131072319' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T02:04:01.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi072-83605-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi072-83605-1"}]': finished 2024-03-26T02:04:01.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:01 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi072-83563-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi072-83563-1"}]': finished 2024-03-26T02:04:01.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:01.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/692566042' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:01.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:01 smithi173 ceph-mon[33839]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T02:04:01.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4045723459' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T02:04:01.917 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:03:58.821181+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name" api_service: [==========] Running 4 tests from 1 test suite. 2024-03-26T02:04:01.917 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: [----------] Global test environment set-up. 2024-03-26T02:04:01.917 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: [----------] 4 tests from LibRadosService 2024-03-26T02:04:01.918 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-03-26T02:04:01.918 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5062 ms) 2024-03-26T02:04:01.918 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-03-26T02:04:01.918 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: [ OK ] LibRadosService.RegisterLate (28 ms) 2024-03-26T02:04:01.918 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-03-26T02:04:01.918 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: cluster: 2024-03-26T02:04:01.918 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: id: c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T02:04:01.918 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: health: HEALTH_WARN 2024-03-26T02:04:01.918 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 7 pool(s) do not have an application enabled 2024-03-26T02:04:01.918 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 2024-03-26T02:04:01.919 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: services: 2024-03-26T02:04:01.919 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-26T02:04:01.919 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-26T02:04:01.919 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-26T02:04:01.919 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-03-26T02:04:01.919 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-26T02:04:01.919 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 2024-03-26T02:04:01.919 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: data: 2024-03-26T02:04:01.919 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: pools: 13 pools, 356 pgs 2024-03-26T02:04:01.919 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: objects: 230 objects, 582 KiB 2024-03-26T02:04:01.919 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: usage: 235 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:04:01.919 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: pgs: 44.944% pgs unknown 2024-03-26T02:04:01.919 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 17.978% pgs not active 2024-03-26T02:04:01.920 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 160 unknown 2024-03-26T02:04:01.920 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 132 active+clean 2024-03-26T02:04:01.920 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 64 creating+peering 2024-03-26T02:04:01.920 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 2024-03-26T02:04:01.920 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: io: 2024-03-26T02:04:01.920 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: client: 639 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-26T02:04:01.920 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 2024-03-26T02:04:01.920 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 2024-03-26T02:04:01.920 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: cluster: 2024-03-26T02:04:01.920 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: id: c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T02:04:01.920 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: health: HEALTH_WARN 2024-03-26T02:04:01.920 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 22 pool(s) do not have an application enabled 2024-03-26T02:04:01.921 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 2024-03-26T02:04:01.921 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: services: 2024-03-26T02:04:01.921 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-26T02:04:01.921 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-26T02:04:01.921 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-26T02:04:01.921 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: foo: 15 portals active (1 hosts, 3 zones) 2024-03-26T02:04:01.921 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-26T02:04:01.921 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 2024-03-26T02:04:01.921 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: data: 2024-03-26T02:04:01.921 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: pools: 28 pools, 836 pgs 2024-03-26T02:04:01.921 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: objects: 230 objects, 582 KiB 2024-03-26T02:04:01.922 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: usage: 235 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:04:01.922 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: pgs: 23.804% pgs unknown 2024-03-26T02:04:01.922 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 56.699% pgs not active 2024-03-26T02:04:01.922 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 474 creating+peering 2024-03-26T02:04:01.922 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 199 unknown 2024-03-26T02:04:01.922 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 163 active+clean 2024-03-26T02:04:01.922 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 2024-03-26T02:04:01.922 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: io: 2024-03-26T02:04:01.922 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: client: 1.3 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-03-26T02:04:01.922 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 2024-03-26T02:04:01.922 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 2024-03-26T02:04:01.922 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: cluster: 2024-03-26T02:04:01.922 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: id: c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T02:04:01.923 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: health: HEALTH_WARN 2024-03-26T02:04:01.923 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 14 pool(s) do not have an application enabled 2024-03-26T02:04:01.923 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 2024-03-26T02:04:01.923 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: services: 2024-03-26T02:04:01.923 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-26T02:04:01.923 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-26T02:04:01.923 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-26T02:04:01.923 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-03-26T02:04:02.182 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: lau: "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:02.183 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:03:58.821274+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82972-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:02.183 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:03:58.821336+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi072-83158-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:02.183 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:03:58.827888+0000 mon.b [INF] from='client.? 172.21.15.72:0/4017972454' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi072-82938-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:04:02.183 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:03:59.433273+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi072-82938-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:04:02.183 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:03:59.482463+0000 mon.b [INF] from='client.? 172.21.15.72:0/4133273429' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:02.183 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:03:59.492281+0000 mon.b [INF] from='client.? 172.21.15.72:0/711058968' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:02.183 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:00.035562+0000 mon.a [INF] from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi072-83605-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi072-83605-1"}]': finished 2024-03-26T02:04:02.183 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:00.035642+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi072-83563-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi072-83563-1"}]': finished 2024-03-26T02:04:02.183 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:00.389542+0000 mon.b [INF] from='client.? 172.21.15.72:0/692566042' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:02.183 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:00.646377+0000 mon.b [INF] from='client.? 172.21.15.72:0/4045723459' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T02:04:02.183 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:01.144922+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi072-82938-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]': finished 2024-03-26T02:04:02.183 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:01.158080+0000 mon.c [INF] from='client.? 172.21.15.72:0/2762249266' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi072-82387-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:02.184 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:01.442141+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi072-82387-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:02.195 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:01.444052+0000 mon.a [INF] from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi072-824925... 2024-03-26T02:04:02.195 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_15_[84628]: listed object 50... 2024-03-26T02:04:02.195 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_15_[84628]: listed object 75... 2024-03-26T02:04:02.195 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_15_[84628]: listed object 100... 2024-03-26T02:04:02.195 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_15_[84628]: listed object 125... 2024-03-26T02:04:02.195 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_15_[84628]: saw 150 objects 2024-03-26T02:04:02.195 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_15_[84628]: shutting down. 2024-03-26T02:04:02.196 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:02.196 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:02.196 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:02.196 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:02.196 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_16_[84629]: starting. 2024-03-26T02:04:02.196 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_16_[84629]: added 25 objects... 2024-03-26T02:04:02.196 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_16_[84629]: added half of the objects 2024-03-26T02:04:02.196 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_16_[84629]: added 50 objects... 2024-03-26T02:04:02.196 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_16_[84629]: added 50 objects 2024-03-26T02:04:02.196 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: process_16_[84629]: shutting down. 2024-03-26T02:04:02.196 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:02.196 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:02.196 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:02.196 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:02.196 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******************************* 2024-03-26T02:04:02.197 INFO:tasks.workunit.client.0.smithi072.stdout: list_parallel: ******* SUCCESS ********** 2024-03-26T02:04:02.204 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-03-26T02:04:02.204 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-03-26T02:04:02.204 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-03-26T02:04:02.204 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-03-26T02:04:02.204 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-03-26T02:04:02.204 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-03-26T02:04:02.204 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-03-26T02:04:02.204 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (289 ms) 2024-03-26T02:04:02.204 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-03-26T02:04:02.204 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (32 ms) 2024-03-26T02:04:02.204 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-03-26T02:04:02.204 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (7 ms) 2024-03-26T02:04:02.205 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-03-26T02:04:02.205 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (36 ms) 2024-03-26T02:04:02.205 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-03-26T02:04:02.205 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (8 ms) 2024-03-26T02:04:02.205 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-03-26T02:04:02.205 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (6 ms) 2024-03-26T02:04:02.205 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-03-26T02:04:02.205 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (4 ms) 2024-03-26T02:04:02.205 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-03-26T02:04:02.205 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (8 ms) 2024-03-26T02:04:02.205 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-03-26T02:04:02.205 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (4 ms) 2024-03-26T02:04:02.205 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-03-26T02:04:02.206 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (4 ms) 2024-03-26T02:04:02.206 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-03-26T02:04:02.206 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (5 ms) 2024-03-26T02:04:02.206 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-03-26T02:04:02.206 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (5 ms) 2024-03-26T02:04:02.206 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-03-26T02:04:02.206 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (13 ms) 2024-03-26T02:04:02.206 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-03-26T02:04:02.206 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (9 ms) 2024-03-26T02:04:02.206 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-03-26T02:04:02.206 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (8 ms) 2024-03-26T02:04:02.206 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-03-26T02:04:02.206 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (3 ms) 2024-03-26T02:04:02.207 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (442 ms total) 2024-03-26T02:04:02.207 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: 2024-03-26T02:04:02.207 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-03-26T02:04:02.207 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (12902 ms total) 2024-03-26T02:04:02.207 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-03-26T02:04:02.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:04:02.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi072-82938-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]': finished 2024-03-26T02:04:02.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[30177]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T02:04:02.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2762249266' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi072-82387-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:02.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:02.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi072-82387-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:02.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi072-82496-10"}]: dispatch 2024-03-26T02:04:02.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-03-26T02:04:02.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-03-26T02:04:02.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-03-26T02:04:02.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-03-26T02:04:02.472 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-03-26T02:04:02.472 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-03-26T02:04:02.472 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-03-26T02:04:02.472 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-03-26T02:04:02.472 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-03-26T02:04:02.472 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-03-26T02:04:02.472 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi072-82496-10"}]: dispatch 2024-03-26T02:04:02.472 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi072-82496-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:02.472 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/116196456' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T02:04:02.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:04:02.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi072-82938-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]': finished 2024-03-26T02:04:02.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[41619]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T02:04:02.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2762249266' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi072-82387-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:02.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:02.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi072-82387-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:02.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi072-82496-10"}]: dispatch 2024-03-26T02:04:02.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-03-26T02:04:02.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-03-26T02:04:02.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-03-26T02:04:02.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-03-26T02:04:02.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-03-26T02:04:02.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-03-26T02:04:02.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-03-26T02:04:02.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-03-26T02:04:02.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-03-26T02:04:02.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-03-26T02:04:02.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi072-82496-10"}]: dispatch 2024-03-26T02:04:02.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi072-82496-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:02.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/116196456' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T02:04:02.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:02 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:04:02.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:02 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi072-82938-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]': finished 2024-03-26T02:04:02.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:02 smithi173 ceph-mon[33839]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T02:04:02.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2762249266' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi072-82387-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:02.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:02.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:02 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi072-82387-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:02.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi072-82496-10"}]: dispatch 2024-03-26T02:04:02.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-03-26T02:04:02.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-03-26T02:04:02.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-03-26T02:04:02.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-03-26T02:04:02.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-03-26T02:04:02.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-03-26T02:04:02.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-03-26T02:04:02.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-03-26T02:04:02.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-03-26T02:04:02.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-03-26T02:04:02.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi072-82496-10"}]: dispatch 2024-03-26T02:04:02.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi072-82496-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:02.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/116196456' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T02:04:03.196 INFO:tasks.workunit.client.0.smithi072.stdout:6-10"}]: dispatch 2024-03-26T02:04:03.196 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:01.621663+0000 mon.c [INF] from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-03-26T02:04:03.196 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:01.622245+0000 mon.c [INF] from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-03-26T02:04:03.197 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:01.622662+0000 mon.c [INF] from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-03-26T02:04:03.197 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:01.623084+0000 mon.c [INF] from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-03-26T02:04:03.197 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:01.623464+0000 mon.c [INF] from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-03-26T02:04:03.197 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:01.623876+0000 mon.c [INF] from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-03-26T02:04:03.197 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:01.624286+0000 mon.c [INF] from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-03-26T02:04:03.197 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:01.624678+0000 mon.c [INF] from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-03-26T02:04:03.197 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:01.625103+0000 mon.c [INF] from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-03-26T02:04:03.197 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:01.625497+0000 mon.c [INF] from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-03-26T02:04:03.197 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:01.687058+0000 mon.a [INF] from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi072-82496-10"}]: dispatch 2024-03-26T02:04:03.197 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:01.687562+0000 mon.a [INF] from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi072-82496-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:03.197 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:02.175789+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:03.197 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:02.179198+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:03.197 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:02.180354+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi072-82387-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:03.197 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:02.180411+0000 mon.a [INF] from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi072-82496-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:03.198 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:02.201237+0000 mon.b [INF] from='client.? 172.21.15.72:0/107770966' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:03.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: pgmap v76: 860 pgs: 124 unknown, 71 creating+peering, 665 active+clean; 610 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 141 KiB/s wr, 584 op/s 2024-03-26T02:04:03.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-03-26T02:04:03.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-03-26T02:04:03.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-03-26T02:04:03.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-03-26T02:04:03.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-03-26T02:04:03.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-03-26T02:04:03.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-03-26T02:04:03.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-03-26T02:04:03.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-03-26T02:04:03.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-03-26T02:04:03.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: 15.8 deep-scrub starts 2024-03-26T02:04:03.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: 15.8 deep-scrub ok 2024-03-26T02:04:03.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:03.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:03.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi072-82387-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:03.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi072-82496-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:03.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/107770966' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:03.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:03.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T02:04:03.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/711058968' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:03.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4133273429' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:03.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi072-82496-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi072-82496-10"}]: dispatch 2024-03-26T02:04:03.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4240982919' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:03.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3769470933' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:03.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: 15.6 deep-scrub starts 2024-03-26T02:04:03.472 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: 15.6 deep-scrub ok 2024-03-26T02:04:03.472 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:03.472 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3769470933' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:03.472 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:03.472 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3769470933' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi072-82723-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:03.472 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi072-82723-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:03.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: pgmap v76: 860 pgs: 124 unknown, 71 creating+peering, 665 active+clean; 610 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 141 KiB/s wr, 584 op/s 2024-03-26T02:04:03.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-03-26T02:04:03.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-03-26T02:04:03.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-03-26T02:04:03.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-03-26T02:04:03.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-03-26T02:04:03.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-03-26T02:04:03.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-03-26T02:04:03.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-03-26T02:04:03.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-03-26T02:04:03.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-03-26T02:04:03.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: 15.8 deep-scrub starts 2024-03-26T02:04:03.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: 15.8 deep-scrub ok 2024-03-26T02:04:03.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:03.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:03.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi072-82387-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:03.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi072-82496-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:03.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/107770966' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:03.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:03.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T02:04:03.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/711058968' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:03.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4133273429' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:03.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi072-82496-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi072-82496-10"}]: dispatch 2024-03-26T02:04:03.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4240982919' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:03.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3769470933' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:03.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: 15.6 deep-scrub starts 2024-03-26T02:04:03.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: 15.6 deep-scrub ok 2024-03-26T02:04:03.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:03.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3769470933' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:03.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:03.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3769470933' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi072-82723-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:03.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:03 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi072-82723-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: pgmap v76: 860 pgs: 124 unknown, 71 creating+peering, 665 active+clean; 610 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 141 KiB/s wr, 584 op/s 2024-03-26T02:04:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-03-26T02:04:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-03-26T02:04:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-03-26T02:04:03.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-03-26T02:04:03.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-03-26T02:04:03.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-03-26T02:04:03.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-03-26T02:04:03.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-03-26T02:04:03.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-03-26T02:04:03.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-03-26T02:04:03.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: 15.8 deep-scrub starts 2024-03-26T02:04:03.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: 15.8 deep-scrub ok 2024-03-26T02:04:03.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:03.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:03.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi072-82387-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:03.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi072-82496-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:03.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/107770966' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:03.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:03.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T02:04:03.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/711058968' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:03.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4133273429' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:03.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi072-82496-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi072-82496-10"}]: dispatch 2024-03-26T02:04:03.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4240982919' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:03.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3769470933' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:03.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: 15.6 deep-scrub starts 2024-03-26T02:04:03.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: 15.6 deep-scrub ok 2024-03-26T02:04:03.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:03.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3769470933' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:03.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:03.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3769470933' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi072-82723-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:03.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:03 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi072-82723-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:04.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[30177]: onexx 2024-03-26T02:04:04.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[30177]: 15.9 deep-scrub starts 2024-03-26T02:04:04.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[30177]: 15.9 deep-scrub ok 2024-03-26T02:04:04.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[30177]: 15.3 deep-scrub starts 2024-03-26T02:04:04.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[30177]: 15.3 deep-scrub ok 2024-03-26T02:04:04.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[30177]: 15.7 deep-scrub starts 2024-03-26T02:04:04.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[30177]: 15.7 deep-scrub ok 2024-03-26T02:04:04.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[30177]: 15.5 deep-scrub starts 2024-03-26T02:04:04.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[30177]: 15.5 deep-scrub ok 2024-03-26T02:04:04.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:04.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T02:04:04.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:04.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:04.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:04.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4240982919' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:04.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[41619]: onexx 2024-03-26T02:04:04.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[41619]: 15.9 deep-scrub starts 2024-03-26T02:04:04.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[41619]: 15.9 deep-scrub ok 2024-03-26T02:04:04.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[41619]: 15.3 deep-scrub starts 2024-03-26T02:04:04.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[41619]: 15.3 deep-scrub ok 2024-03-26T02:04:04.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[41619]: 15.7 deep-scrub starts 2024-03-26T02:04:04.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[41619]: 15.7 deep-scrub ok 2024-03-26T02:04:04.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[41619]: 15.5 deep-scrub starts 2024-03-26T02:04:04.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[41619]: 15.5 deep-scrub ok 2024-03-26T02:04:04.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:04.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T02:04:04.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:04.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:04.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:04.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4240982919' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:04.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:04 smithi173 ceph-mon[33839]: onexx 2024-03-26T02:04:04.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:04 smithi173 ceph-mon[33839]: 15.9 deep-scrub starts 2024-03-26T02:04:04.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:04 smithi173 ceph-mon[33839]: 15.9 deep-scrub ok 2024-03-26T02:04:04.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:04 smithi173 ceph-mon[33839]: 15.3 deep-scrub starts 2024-03-26T02:04:04.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:04 smithi173 ceph-mon[33839]: 15.3 deep-scrub ok 2024-03-26T02:04:04.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:04 smithi173 ceph-mon[33839]: 15.7 deep-scrub starts 2024-03-26T02:04:04.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:04 smithi173 ceph-mon[33839]: 15.7 deep-scrub ok 2024-03-26T02:04:04.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:04 smithi173 ceph-mon[33839]: 15.5 deep-scrub starts 2024-03-26T02:04:04.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:04 smithi173 ceph-mon[33839]: 15.5 deep-scrub ok 2024-03-26T02:04:04.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:04 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:04.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:04 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T02:04:04.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:04 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:04.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:04 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:04.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:04 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:04.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:04 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4240982919' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:05.437 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:04:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:04:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:04:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: 15.0 deep-scrub starts 2024-03-26T02:04:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: 15.0 deep-scrub ok 2024-03-26T02:04:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi072-82723-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3769470933' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi072-82723-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:05.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3366064809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi072-83563-1"}]: dispatch 2024-03-26T02:04:05.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1381361081' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi072-82404-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:05.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:05.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: pgmap v79: 852 pgs: 363 unknown, 39 creating+peering, 450 active+clean; 586 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 22 KiB/s wr, 413 op/s 2024-03-26T02:04:05.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/692566042' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:05.722 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: 15.4 deep-scrub starts 2024-03-26T02:04:05.722 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: 15.4 deep-scrub ok 2024-03-26T02:04:05.722 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: 15.1 deep-scrub starts 2024-03-26T02:04:05.722 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: 15.1 deep-scrub ok 2024-03-26T02:04:05.722 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: 15.2 deep-scrub starts 2024-03-26T02:04:05.722 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: 15.2 deep-scrub ok 2024-03-26T02:04:05.723 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:05.723 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T02:04:05.723 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4133273429' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi072-82429-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:05.723 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/711058968' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:05.723 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi072-82723-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:05.723 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi072-83563-1"}]: dispatch 2024-03-26T02:04:05.724 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-26T02:04:05.724 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4045723459' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T02:04:05.724 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: twoxx 2024-03-26T02:04:05.724 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi072-82404-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:05.725 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: 15.0 deep-scrub starts 2024-03-26T02:04:05.725 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: 15.0 deep-scrub ok 2024-03-26T02:04:05.725 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi072-82723-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:05.725 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3769470933' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi072-82723-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:05.725 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3366064809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi072-83563-1"}]: dispatch 2024-03-26T02:04:05.726 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1381361081' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi072-82404-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:05.726 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:05.726 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: pgmap v79: 852 pgs: 363 unknown, 39 creating+peering, 450 active+clean; 586 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 22 KiB/s wr, 413 op/s 2024-03-26T02:04:05.726 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/692566042' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:05.726 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: 15.4 deep-scrub starts 2024-03-26T02:04:05.726 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: 15.4 deep-scrub ok 2024-03-26T02:04:05.727 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: 15.1 deep-scrub starts 2024-03-26T02:04:05.727 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: 15.1 deep-scrub ok 2024-03-26T02:04:05.727 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: 15.2 deep-scrub starts 2024-03-26T02:04:05.727 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: 15.2 deep-scrub ok 2024-03-26T02:04:05.727 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:05.728 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T02:04:05.728 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4133273429' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi072-82429-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:05.728 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/711058968' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:05.728 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi072-82723-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:05.728 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi072-83563-1"}]: dispatch 2024-03-26T02:04:05.728 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-26T02:04:05.729 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4045723459' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T02:04:05.729 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: twoxx 2024-03-26T02:04:05.729 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi072-82404-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:05.729 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi072-82496-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi072-82496-10"}]': finished 2024-03-26T02:04:05.730 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:05.730 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2195813754' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:05.730 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T02:04:05.730 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:05.730 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:05.731 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/692566042' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:05.731 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi072-82429-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:05.731 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2586181871' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:05.731 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:05.731 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T02:04:05.731 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:05.732 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:05.732 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2586181871' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:05.732 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:05.732 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:05.732 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2586181871' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi072-82604-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:05.732 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi072-82604-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:05.733 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:05.733 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi072-82496-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi072-82496-10"}]': finished 2024-03-26T02:04:05.733 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:05.733 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2195813754' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:05.734 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T02:04:05.734 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:05.734 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:05.734 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/692566042' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:05.734 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi072-82429-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:05.734 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2586181871' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:05.735 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:05.735 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T02:04:05.735 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:05.735 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:05.736 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2586181871' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:05.736 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:05.736 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:05.736 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2586181871' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi072-82604-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:05.736 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi072-82604-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:05.736 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:05.746 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:02.461920+0000 mon.b [INF] from='client.? 172.21.15.72:0/711058968' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi072-82437-2 open_pools_parallel: process_1_[83258]: starting. 2024-03-26T02:04:05.747 INFO:tasks.workunit.client.0.smithi072.stdout: open_pools_parallel: process_1_[83258]: creating pool ceph_test_rados_open_pools_parallel.smithi072-83190 2024-03-26T02:04:05.747 INFO:tasks.workunit.client.0.smithi072.stdout: open_pools_parallel: process_1_[83258]: created object 0... 2024-03-26T02:04:05.747 INFO:tasks.workunit.client.0.smithi072.stdout: open_pools_parallel: process_1_[83258]: created object 25... 2024-03-26T02:04:05.747 INFO:tasks.workunit.client.0.smithi072.stdout: open_pools_parallel: process_1_[83258]: created object 49... 2024-03-26T02:04:05.747 INFO:tasks.workunit.client.0.smithi072.stdout: open_pools_parallel: process_1_[83258]: finishing. 2024-03-26T02:04:05.747 INFO:tasks.workunit.client.0.smithi072.stdout: open_pools_parallel: process_1_[83258]: shutting down. 2024-03-26T02:04:05.748 INFO:tasks.workunit.client.0.smithi072.stdout: open_pools_parallel: process_2_[83260]: starting. 2024-03-26T02:04:05.748 INFO:tasks.workunit.client.0.smithi072.stdout: open_pools_parallel: process_2_[83260]: rados_pool_create. 2024-03-26T02:04:05.748 INFO:tasks.workunit.client.0.smithi072.stdout: open_pools_parallel: process_2_[83260]: rados_ioctx_create. 2024-03-26T02:04:05.748 INFO:tasks.workunit.client.0.smithi072.stdout: open_pools_parallel: process_2_[83260]: shutting down. 2024-03-26T02:04:05.748 INFO:tasks.workunit.client.0.smithi072.stdout: open_pools_parallel: ******************************* 2024-03-26T02:04:05.748 INFO:tasks.workunit.client.0.smithi072.stdout: open_pools_parallel: process_3_[84680]: starting. 2024-03-26T02:04:05.749 INFO:tasks.workunit.client.0.smithi072.stdout: open_pools_parallel: process_3_[84680]: creating pool ceph_test_rados_open_pools_parallel.smithi072-83190 2024-03-26T02:04:05.749 INFO:tasks.workunit.client.0.smithi072.stdout: open_pools_parallel: process_3_[84680]: created object 0... 2024-03-26T02:04:05.749 INFO:tasks.workunit.client.0.smithi072.stdout: open_pools_parallel: process_3_[84680]: created object 25... 2024-03-26T02:04:05.750 INFO:tasks.workunit.client.0.smithi072.stdout: open_pools_parallel: process_3_[84680]: created object 49... 2024-03-26T02:04:05.750 INFO:tasks.workunit.client.0.smithi072.stdout: open_pools_parallel: process_3_[84680]: finishing. 2024-03-26T02:04:05.750 INFO:tasks.workunit.client.0.smithi072.stdout: open_pools_parallel: process_3_[84680]: shutting down. 2024-03-26T02:04:05.750 INFO:tasks.workunit.client.0.smithi072.stdout: open_pools_parallel: ******************************* 2024-03-26T02:04:05.750 INFO:tasks.workunit.client.0.smithi072.stdout: open_pools_parallel: process_4_[84681]: starting. 2024-03-26T02:04:05.751 INFO:tasks.workunit.client.0.smithi072.stdout: open_pools_parallel: process_4_[84681]: rados_pool_create. 2024-03-26T02:04:05.751 INFO:tasks.workunit.client.0.smithi072.stdout: open_pools_parallel: process_4_[84681]: rados_ioctx_create. 2024-03-26T02:04:05.751 INFO:tasks.workunit.client.0.smithi072.stdout: open_pools_parallel: process_4_[84681]: shutting down. 2024-03-26T02:04:05.751 INFO:tasks.workunit.client.0.smithi072.stdout: open_pools_parallel: ******************************* 2024-03-26T02:04:05.751 INFO:tasks.workunit.client.0.smithi072.stdout: open_pools_parallel: ******************************* 2024-03-26T02:04:05.751 INFO:tasks.workunit.client.0.smithi072.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-03-26T02:04:05.752 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: process_1_[83301]: starting. 2024-03-26T02:04:05.752 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: process_1_[83301]: creating pool ceph_test_rados_delete_pools_parallel.smithi072-83245 2024-03-26T02:04:05.752 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: process_1_[83301]: created object 0... 2024-03-26T02:04:05.752 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: process_1_[83301]: created object 25... 2024-03-26T02:04:05.752 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: process_1_[83301]: created object 49... 2024-03-26T02:04:05.752 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: process_1_[83301]: finishing. 2024-03-26T02:04:05.752 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: process_1_[83301]: shutting down. 2024-03-26T02:04:05.752 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: process_2_[83302]: starting. 2024-03-26T02:04:05.753 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: process_2_[83302]: deleting pool ceph_test_rados_delete_pools_parallel.smithi072-83245 2024-03-26T02:04:05.753 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: process_2_[83302]: shutting down. 2024-03-26T02:04:05.753 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: ******************************* 2024-03-26T02:04:05.753 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: process_3_[84677]: starting. 2024-03-26T02:04:05.753 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: process_3_[84677]: creating pool ceph_test_rados_delete_pools_parallel.smithi072-83245 2024-03-26T02:04:05.753 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: process_3_[84677]: created object 0... 2024-03-26T02:04:05.754 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: process_3_[84677]: created object 25... 2024-03-26T02:04:05.754 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: process_3_[84677]: created object 49... 2024-03-26T02:04:05.754 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: process_3_[84677]: finishing. 2024-03-26T02:04:05.754 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: process_3_[84677]: shutting down. 2024-03-26T02:04:05.754 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: ******************************* 2024-03-26T02:04:05.754 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: process_5_[84679]: starting. 2024-03-26T02:04:05.754 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: process_5_[84679]: listing objects. 2024-03-26T02:04:05.754 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: process_5_[84679]: listed object 0... 2024-03-26T02:04:05.754 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: process_5_[84679]: listed object 25... 2024-03-26T02:04:05.754 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: process_5_[84679]: saw 50 objects 2024-03-26T02:04:05.754 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: process_5_[84679]: shutting down. 2024-03-26T02:04:05.754 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: ******************************* 2024-03-26T02:04:05.754 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: process_4_[84678]: starting. 2024-03-26T02:04:05.755 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: process_4_[84678]: deleting pool ceph_test_rados_delete_pools_parallel.smithi072-83245 2024-03-26T02:04:05.755 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: process_4_[84678]: shutting down. 2024-03-26T02:04:05.755 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: ******************************* 2024-03-26T02:04:05.755 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: ******************************* 2024-03-26T02:04:05.755 INFO:tasks.workunit.client.0.smithi072.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-03-26T02:04:05.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: 15.0 deep-scrub starts 2024-03-26T02:04:05.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: 15.0 deep-scrub ok 2024-03-26T02:04:05.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi072-82723-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:05.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3769470933' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi072-82723-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:05.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3366064809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi072-83563-1"}]: dispatch 2024-03-26T02:04:05.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1381361081' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi072-82404-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:05.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:05.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: pgmap v79: 852 pgs: 363 unknown, 39 creating+peering, 450 active+clean; 586 KiB data, 263 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 22 KiB/s wr, 413 op/s 2024-03-26T02:04:05.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/692566042' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:05.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: 15.4 deep-scrub starts 2024-03-26T02:04:05.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: 15.4 deep-scrub ok 2024-03-26T02:04:05.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: 15.1 deep-scrub starts 2024-03-26T02:04:05.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: 15.1 deep-scrub ok 2024-03-26T02:04:05.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: 15.2 deep-scrub starts 2024-03-26T02:04:05.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: 15.2 deep-scrub ok 2024-03-26T02:04:05.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:05.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T02:04:05.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4133273429' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi072-82429-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:05.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/711058968' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:05.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi072-82723-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:05.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi072-83563-1"}]: dispatch 2024-03-26T02:04:05.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-26T02:04:05.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4045723459' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T02:04:05.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: twoxx 2024-03-26T02:04:05.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi072-82404-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:05.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi072-82496-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi072-82496-10"}]': finished 2024-03-26T02:04:05.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:05.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2195813754' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:05.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T02:04:05.816 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:05.816 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:05.816 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/692566042' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:05.816 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi072-82429-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:05.816 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2586181871' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:05.816 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:05.816 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T02:04:05.816 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:05.816 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:05.816 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2586181871' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:05.816 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:05.816 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:05.816 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2586181871' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi072-82604-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:05.816 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi072-82604-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:05.817 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:06.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:06 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-26T02:04:06.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:06 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi072-83563-1"}]': finished 2024-03-26T02:04:06.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:06 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi072-82404-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:06.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:06 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi072-82429-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:06.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:06 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:06.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:06 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:06.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:06 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:06.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:06 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi072-82604-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:06.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3366064809' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi072-83563-1"}]: dispatch 2024-03-26T02:04:06.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/692566042' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi072-82877-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:06.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4133273429' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi072-82429-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:06.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/711058968' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi072-82437-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:06.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2586181871' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi072-82604-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:06.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:06 smithi173 ceph-mon[33839]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T02:04:06.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/107770966' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:06.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:06.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4017972454' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:04:06.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:06 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi072-83563-1"}]: dispatch 2024-03-26T02:04:06.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:06 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi072-82604-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:06.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:06 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi072-82877-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:06.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi072-83605-1"}]: dispatch 2024-03-26T02:04:06.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:06 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi072-82429-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:06.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:06 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi072-82437-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:06.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:06 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:04:06.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:06 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:04:06.815 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:06 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:06.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-26T02:04:06.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi072-83563-1"}]': finished 2024-03-26T02:04:06.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi072-82404-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:06.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi072-82429-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:06.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:06.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:06.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:06.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi072-82604-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:06.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3366064809' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi072-83563-1"}]: dispatch 2024-03-26T02:04:06.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/692566042' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi072-82877-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:06.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4133273429' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi072-82429-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:06.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/711058968' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi072-82437-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:06.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2586181871' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi072-82604-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:06.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[41619]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T02:04:06.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/107770966' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:06.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:06.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4017972454' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:04:06.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi072-83563-1"}]: dispatch 2024-03-26T02:04:06.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi072-82604-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:06.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi072-82877-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:06.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi072-83605-1"}]: dispatch 2024-03-26T02:04:06.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi072-82429-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:06.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi072-82437-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:06.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:04:06.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:04:06.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:06.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-26T02:04:06.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi072-83563-1"}]': finished 2024-03-26T02:04:06.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi072-82404-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:06.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi072-82429-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:06.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:06.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:06.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:06.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi072-82604-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:06.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3366064809' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi072-83563-1"}]: dispatch 2024-03-26T02:04:06.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/692566042' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi072-82877-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:06.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4133273429' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi072-82429-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:06.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/711058968' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi072-82437-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:06.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2586181871' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi072-82604-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:06.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[30177]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T02:04:06.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/107770966' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:06.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:06.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4017972454' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:04:06.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi072-83563-1"}]: dispatch 2024-03-26T02:04:06.973 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi072-82604-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:06.973 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi072-82877-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:06.973 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi072-83605-1"}]: dispatch 2024-03-26T02:04:06.973 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi072-82429-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:06.973 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi072-82437-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:06.973 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:04:06.973 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:04:06.973 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:06 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:07.442 INFO:tasks.workunit.client.0.smithi072.stdout:2"}]: dispatch 2024-03-26T02:04:07.442 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:02.462106+0000 mon.b [INF] from='client.? 172.21.15.72:0/4133273429' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:07.442 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:02.463264+0000 mon.a [INF] from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi072-82496-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi072-82496-10"}]: dispatch 2024-03-26T02:04:07.442 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:02.464833+0000 mon.a [INF] from='client.? 172.21.15.72:0/4240982919' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:07.442 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:02.476386+0000 mon.c [INF] from='client.? 172.21.15.72:0/3769470933' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:07.442 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:02.712034+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:07.442 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:02.712617+0000 mon.c [INF] from='client.? 172.21.15.72:0/3769470933' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:07.442 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:02.950544+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:07.443 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:02.951108+0000 mon.c [INF] from='client.? 172.21.15.72:0/3769470933' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi072-82723-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:07.443 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:02.951520+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi072-82723-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:07.443 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:00.646557+0000 client.admin [INF] onexx 2024-03-26T02:04:07.443 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:03.191626+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:07.443 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:03.192082+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T02:04:07.443 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:03.192549+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:07.443 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:03.192744+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:07.443 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:03.192856+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:07.443 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:03.205192+0000 mon.a [INF] from='client.? 172.21.15.72:0/4240982919' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:07.443 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:05.437613+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-26T02:04:07.443 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:05.447358+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi072-83563-1"}]': finished 2024-03-26T02:04:07.443 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:05.447427+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi072-82404-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:07.443 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:05.447499+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi072-82429-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:07.443 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:05.447546+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:07.444 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:05.447588+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:07.444 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:05.447628+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:07.444 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:05.447671+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi072-82604-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:07.444 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:05.451998+0000 mon.b [INF] from='client.? 172.21.15.72:0/3366064809' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi072-83563-1"}]: dispatch 2024-03-26T02:04:07.444 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:05.452320+0000 mon.b [INF] from='client.? 172.21.15.72:0/692566042' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi072-82877-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:07.444 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:05.452491+0000 mon.b [INF] from='client.? 172.21.15.72:0/4133273429' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi072-82429-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:07.444 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:05.452855+0000 mon.b [INF] from='client.? 172.21.15.72:0/711058968' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi072-82437-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:07.444 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:05.454429+0000 mon.c [INF] from='client.? 172.21.15.72:0/2586181871' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi072-82604-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:07.444 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:05.741006+0000 mon.b [INF] from='client.? 172.21.15.72:0/107770966' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:07.444 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:05.741556+0000 mon.c [INF] from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:07.480 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:05.742798+0000 mon.b cmd: Running main() from gmock_main.cc 2024-03-26T02:04:07.481 INFO:tasks.workunit.client.0.smithi072.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-03-26T02:04:07.481 INFO:tasks.workunit.client.0.smithi072.stdout: cmd: [----------] Global test environment set-up. 2024-03-26T02:04:07.481 INFO:tasks.workunit.client.0.smithi072.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-03-26T02:04:07.481 INFO:tasks.workunit.client.0.smithi072.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-03-26T02:04:07.481 INFO:tasks.workunit.client.0.smithi072.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (10021 ms) 2024-03-26T02:04:07.481 INFO:tasks.workunit.client.0.smithi072.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-03-26T02:04:07.481 INFO:tasks.workunit.client.0.smithi072.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (3031 ms) 2024-03-26T02:04:07.481 INFO:tasks.workunit.client.0.smithi072.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-03-26T02:04:07.481 INFO:tasks.workunit.client.0.smithi072.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (5013 ms) 2024-03-26T02:04:07.481 INFO:tasks.workunit.client.0.smithi072.stdout: cmd: [----------] 3 tests from NeoRadosCmd (18065 ms total) 2024-03-26T02:04:07.481 INFO:tasks.workunit.client.0.smithi072.stdout: cmd: 2024-03-26T02:04:07.481 INFO:tasks.workunit.client.0.smithi072.stdout: cmd: [----------] Global test environment tear-down 2024-03-26T02:04:07.482 INFO:tasks.workunit.client.0.smithi072.stdout: cmd: [==========] 3 tests from 1 test suite ran. (18065 ms total) 2024-03-26T02:04:07.482 INFO:tasks.workunit.client.0.smithi072.stdout: cmd: [ PASSED ] 3 tests. 2024-03-26T02:04:07.489 INFO:tasks.workunit.client.0.smithi072.stdout:[INF] from='client.? 172.21.15.72:0/4017972454' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:04:07.489 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:05.744281+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi072-83563-1"}]: dispatch 2024-03-26T02:04:07.489 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:05.744666+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi072-82604-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:07.489 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:05.748069+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi072-82877-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:07.489 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:05.748754+0000 mon.a [INF] from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi072-83605-1"}]: dispatch 2024-03-26T02:04:07.489 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:05.748976+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi072-82429-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:07.489 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:05.749780+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi072-82437-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:07.490 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:06.000171+0000 mon.a [INF] from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:04:07.490 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:06.006000+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:07.490 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:06.490085+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi072-82723-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi072-82723-7"}]': finished 2024-03-26T02:04:07.490 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:06.490185+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi072-83563-1"}]': finished 2024-03-26T02:04:07.490 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:06.490254+0000 mon.a [INF] from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi072-83605-1"}]': finished 2024-03-26T02:04:07.490 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:06.490292+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:07.490 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:06.527971+0000 mon.c [INF] from='client.? 172.21.15.72:0/964239072' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi072-83563-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:07.490 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:06.591835+0000 mon.c [INF] from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:07 smithi173 ceph-mon[33839]: pgmap v82: 844 pgs: 96 unknown, 264 creating+peering, 484 active+clean; 145 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 46 op/s 2024-03-26T02:04:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:07 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi072-82723-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi072-82723-7"}]': finished 2024-03-26T02:04:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:07 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi072-83563-1"}]': finished 2024-03-26T02:04:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi072-83605-1"}]': finished 2024-03-26T02:04:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:07 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/964239072' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi072-83563-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-26T02:04:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:07.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4045723459' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T02:04:07.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1381361081' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi072-82404-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:07.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:07 smithi173 ceph-mon[33839]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T02:04:07.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1402942902' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi072-82387-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:07.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1584093742' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:07.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[30177]: pgmap v82: 844 pgs: 96 unknown, 264 creating+peering, 484 active+clean; 145 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 46 op/s 2024-03-26T02:04:07.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:07.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi072-82723-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi072-82723-7"}]': finished 2024-03-26T02:04:07.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi072-83563-1"}]': finished 2024-03-26T02:04:07.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi072-83605-1"}]': finished 2024-03-26T02:04:07.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:07.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/964239072' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi072-83563-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:07.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-26T02:04:07.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:07.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:07.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4045723459' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T02:04:07.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1381361081' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi072-82404-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:07.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[30177]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T02:04:07.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1402942902' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi072-82387-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:07.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1584093742' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:07.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[41619]: pgmap v82: 844 pgs: 96 unknown, 264 creating+peering, 484 active+clean; 145 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 46 op/s 2024-03-26T02:04:07.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:07.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi072-82723-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi072-82723-7"}]': finished 2024-03-26T02:04:07.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi072-83563-1"}]': finished 2024-03-26T02:04:07.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi072-83605-1"}]': finished 2024-03-26T02:04:07.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:07.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/964239072' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi072-83563-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:07.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-26T02:04:07.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:07.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:07.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4045723459' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T02:04:07.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1381361081' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi072-82404-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:07.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[41619]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T02:04:07.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1402942902' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi072-82387-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:07.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1584093742' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:08.493 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:07.442161+0000 mon.b [INF] from='client.? 172.21.15.72:0/4045723459' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T02:04:08.493 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:07.475923+0000 mon.b [INF] from='client.? 172.21.15.72:0/1381361081' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi072-82404-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:08.493 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:07.476732+0000 mon.c [INF] from='client.? 172.21.15.72:0/1402942902' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi072-82387-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:08.493 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:07.478462+0000 mon.c [INF] from='client.? 172.21.15.72:0/1584093742' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:08.493 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:07.480256+0000 mon.a [INF] from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi072-83605-1"}]: dispatch 2024-03-26T02:04:08.494 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:07.480621+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi072-83563-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:08.494 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:07.498662+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:08.494 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:07.499207+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi072-82387-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:08.494 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:07.499635+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:08.494 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:07.499914+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi072-82604-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi072-82604-10"}]': finished 2024-03-26T02:04:08.494 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:07.500052+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi072-82877-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi072-82877-7"}]': finished 2024-03-26T02:04:08.494 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:07.500134+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi072-82429-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi072-82429-16"}]': finished 2024-03-26T02:04:08.494 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:07.500264+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi072-82437-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi072-82437-22"}]': finished 2024-03-26T02:04:08.494 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:07.500350+0000 mon.a [INF] from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi072-83605-1"}]': finished 2024-03-26T02:04:08.494 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:07.500421+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi072-83563-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:08.662 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:07.509412+0000 mon.c [INF] from='cli api_io_pp: Running main() from gmock_main.cc 2024-03-26T02:04:08.662 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-03-26T02:04:08.662 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [----------] Global test environment set-up. 2024-03-26T02:04:08.662 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-03-26T02:04:08.662 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: seed 82437 2024-03-26T02:04:08.662 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-03-26T02:04:08.662 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-03-26T02:04:08.663 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-03-26T02:04:08.663 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (122 ms) 2024-03-26T02:04:08.663 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-03-26T02:04:08.663 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (7 ms) 2024-03-26T02:04:08.663 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-03-26T02:04:08.663 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (5 ms) 2024-03-26T02:04:08.663 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-03-26T02:04:08.663 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (5 ms) 2024-03-26T02:04:08.663 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-03-26T02:04:08.663 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (2 ms) 2024-03-26T02:04:08.663 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-03-26T02:04:08.663 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (2 ms) 2024-03-26T02:04:08.663 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-03-26T02:04:08.663 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (3 ms) 2024-03-26T02:04:08.663 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-03-26T02:04:08.663 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (5 ms) 2024-03-26T02:04:08.664 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-03-26T02:04:08.664 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (3 ms) 2024-03-26T02:04:08.664 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-03-26T02:04:08.664 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (3 ms) 2024-03-26T02:04:08.664 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-03-26T02:04:08.664 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (6 ms) 2024-03-26T02:04:08.664 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-03-26T02:04:08.664 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (5 ms) 2024-03-26T02:04:08.664 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-03-26T02:04:08.664 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (2 ms) 2024-03-26T02:04:08.664 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-03-26T02:04:08.664 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (3 ms) 2024-03-26T02:04:08.664 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-03-26T02:04:08.664 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (8 ms) 2024-03-26T02:04:08.664 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-26T02:04:08.665 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (4 ms) 2024-03-26T02:04:08.665 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-03-26T02:04:08.665 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (3 ms) 2024-03-26T02:04:08.665 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-03-26T02:04:08.665 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (4 ms) 2024-03-26T02:04:08.665 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-03-26T02:04:08.665 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (4 ms) 2024-03-26T02:04:08.665 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (196 ms total) 2024-03-26T02:04:08.665 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: 2024-03-26T02:04:08.665 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-03-26T02:04:08.665 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-03-26T02:04:08.665 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1115 ms) 2024-03-26T02:04:08.665 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-03-26T02:04:08.665 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (19 ms) 2024-03-26T02:04:08.665 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-03-26T02:04:08.665 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (5 ms) 2024-03-26T02:04:08.666 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-03-26T02:04:08.666 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (4 ms) 2024-03-26T02:04:08.666 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-03-26T02:04:08.666 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (5 ms) 2024-03-26T02:04:08.792 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] L cls: Running main() from gmock_main.cc 2024-03-26T02:04:08.792 INFO:tasks.workunit.client.0.smithi072.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-03-26T02:04:08.792 INFO:tasks.workunit.client.0.smithi072.stdout: cls: [----------] Global test environment set-up. 2024-03-26T02:04:08.793 INFO:tasks.workunit.client.0.smithi072.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-03-26T02:04:08.793 INFO:tasks.workunit.client.0.smithi072.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-03-26T02:04:08.793 INFO:tasks.workunit.client.0.smithi072.stdout: cls: [ OK ] NeoRadosCls.DNE (10973 ms) 2024-03-26T02:04:08.793 INFO:tasks.workunit.client.0.smithi072.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-03-26T02:04:08.793 INFO:tasks.workunit.client.0.smithi072.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (8443 ms) 2024-03-26T02:04:08.793 INFO:tasks.workunit.client.0.smithi072.stdout: cls: [----------] 2 tests from NeoRadosCls (19416 ms total) 2024-03-26T02:04:08.793 INFO:tasks.workunit.client.0.smithi072.stdout: cls: 2024-03-26T02:04:08.793 INFO:tasks.workunit.client.0.smithi072.stdout: cls: [----------] Global test environment tear-down 2024-03-26T02:04:08.793 INFO:tasks.workunit.client.0.smithi072.stdout: cls: [==========] 2 tests from 1 test suite ran. (19416 ms total) 2024-03-26T02:04:08.793 INFO:tasks.workunit.client.0.smithi072.stdout: cls: [ PASSED ] 2 tests. 2024-03-26T02:04:08.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:08 smithi173 ceph-mon[33839]: pgmap v84: 820 pgs: 168 unknown, 168 creating+peering, 484 active+clean; 145 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 35 MiB/s wr, 127 op/s 2024-03-26T02:04:08.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi072-83605-1"}]: dispatch 2024-03-26T02:04:08.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi072-83563-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:08.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:08.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi072-82387-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:08.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:08.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi072-82604-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi072-82604-10"}]': finished 2024-03-26T02:04:08.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi072-82877-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi072-82877-7"}]': finished 2024-03-26T02:04:08.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi072-82429-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi072-82429-16"}]': finished 2024-03-26T02:04:08.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi072-82437-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi072-82437-22"}]': finished 2024-03-26T02:04:08.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi072-83605-1"}]': finished 2024-03-26T02:04:08.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi072-83563-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:08.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:08 smithi173 ceph-mon[33839]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T02:04:08.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/964239072' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi072-83563-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi072-83563-2"}]: dispatch 2024-03-26T02:04:08.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi072-83563-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi072-83563-2"}]: dispatch 2024-03-26T02:04:08.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1118612117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi072-83605-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:08.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi072-83605-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:08.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:08.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:04:08.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[30177]: pgmap v84: 820 pgs: 168 unknown, 168 creating+peering, 484 active+clean; 145 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 35 MiB/s wr, 127 op/s 2024-03-26T02:04:08.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi072-83605-1"}]: dispatch 2024-03-26T02:04:08.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi072-83563-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:08.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:08.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi072-82387-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:08.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:08.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi072-82604-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi072-82604-10"}]': finished 2024-03-26T02:04:08.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi072-82877-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi072-82877-7"}]': finished 2024-03-26T02:04:08.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi072-82429-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi072-82429-16"}]': finished 2024-03-26T02:04:08.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi072-82437-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi072-82437-22"}]': finished 2024-03-26T02:04:08.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi072-83605-1"}]': finished 2024-03-26T02:04:08.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi072-83563-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:08.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[30177]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T02:04:08.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/964239072' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi072-83563-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi072-83563-2"}]: dispatch 2024-03-26T02:04:08.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi072-83563-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi072-83563-2"}]: dispatch 2024-03-26T02:04:08.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1118612117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi072-83605-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:08.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi072-83605-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:08.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:08.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:04:08.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[41619]: pgmap v84: 820 pgs: 168 unknown, 168 creating+peering, 484 active+clean; 145 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 35 MiB/s wr, 127 op/s 2024-03-26T02:04:08.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi072-83605-1"}]: dispatch 2024-03-26T02:04:08.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi072-83563-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:08.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:08.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi072-82387-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:08.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:08.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi072-82604-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi072-82604-10"}]': finished 2024-03-26T02:04:08.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi072-82877-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi072-82877-7"}]': finished 2024-03-26T02:04:08.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi072-82429-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi072-82429-16"}]': finished 2024-03-26T02:04:08.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi072-82437-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi072-82437-22"}]': finished 2024-03-26T02:04:08.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1089743283' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi072-83605-1"}]': finished 2024-03-26T02:04:08.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi072-83563-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:08.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[41619]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T02:04:08.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/964239072' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi072-83563-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi072-83563-2"}]: dispatch 2024-03-26T02:04:08.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi072-83563-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi072-83563-2"}]: dispatch 2024-03-26T02:04:08.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1118612117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi072-83605-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:08.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi072-83605-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:08.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:08.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:04:09.810 INFO:tasks.workunit.client.0.smithi072.stdout:ent.? 172.21.15.72:0/964239072' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi072-83563-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi072-83563-2"}]: dispatch 2024-03-26T02:04:09.810 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.003862+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi072-83563-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi072-83563-2"}]: dispatch 2024-03-26T02:04:09.810 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.008661+0000 mon.c [INF] from='client.? 172.21.15.72:0/1118612117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi072-83605-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:09.811 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.249056+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi072-83605-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:09.811 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.252009+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:09.811 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.252314+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:04:09.811 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:07.442376+0000 client.admin [INF] threexx 2024-03-26T02:04:09.811 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.496043+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T02:04:09.811 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.496191+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi072-82404-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:09.811 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.504126+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:09.811 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.504190+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi072-82387-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:09.811 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.504245+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:09.811 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.504289+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi072-83605-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:09.811 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.504345+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:09.811 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.504391+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]': finished 2024-03-26T02:04:09.812 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.504449+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi072-82404-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:09.812 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.510136+0000 mon.c [INF] from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app2"}]: dispatch 2024-03-26T02:04:09.812 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.510657+0000 mon.c [INF] from='client.? 172.21.15.72:0/1118612117' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi072-83605-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi072-83605-2"}]: dispatch 2024-03-26T02:04:09.812 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.511827+0000 mon.c [INF] from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:09.812 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.514486+0000 mon.b [INF] from='client.? 172.21.15.72:0/4017972454' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:04:09.812 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.515469+0000 mon.b [INF] from='client.? 172.21.15.72:0/1381361081' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi072-82404-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T02:04:09.812 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.515899+0000 mon.b [INF] from='client.? 172.21.15.72:0/107770966' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-4", "tierpool": "test-rados-api-smithi072-82692-4-cache"}]: dispatch 2024-03-26T02:04:09.812 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.788199+0000 mon.c [INF] from='client.? 172.21.15.72:0/3681383293' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:09.812 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.789857+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:04:09.812 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.790372+0000 mon.a [INF] from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi072-82496-10"}]: dispatch 2024-03-26T02:04:09.812 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.790626+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi072-82404-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T02:04:09.812 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.791654+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-4", "tierpool": "test-rados-api-smithi072-82692-4-cache"}]: dispatch 2024-03-26T02:04:09.812 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.791984+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi072-83605-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi072-83605-2"}]: dispatch 2024-03-26T02:04:09.813 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.792587+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:09.813 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.794302+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:09.813 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:08.822757+0000 mon.c [INF] from='client.? 172.21.15.72:0/351269750' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:10.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: threexx 2024-03-26T02:04:10.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:10.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T02:04:10.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi072-82404-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:10.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:10.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi072-82387-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:10.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:10.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi072-83605-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:10.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:10.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]': finished 2024-03-26T02:04:10.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi072-82404-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:10.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app2"}]: dispatch 2024-03-26T02:04:10.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1118612117' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi072-83605-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi072-83605-2"}]: dispatch 2024-03-26T02:04:10.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:10.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4017972454' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:04:10.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1381361081' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi072-82404-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T02:04:10.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/107770966' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-4", "tierpool": "test-rados-api-smithi072-82692-4-cache"}]: dispatch 2024-03-26T02:04:10.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T02:04:10.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3681383293' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:10.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:04:10.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi072-82496-10"}]: dispatch 2024-03-26T02:04:10.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi072-82404-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T02:04:10.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-4", "tierpool": "test-rados-api-smithi072-82692-4-cache"}]: dispatch 2024-03-26T02:04:10.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi072-83605-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi072-83605-2"}]: dispatch 2024-03-26T02:04:10.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:10.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:10.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/351269750' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:10.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:10.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: threexx 2024-03-26T02:04:10.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:10.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T02:04:10.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi072-82404-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:10.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:10.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi072-82387-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:10.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:10.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi072-83605-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:10.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:10.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]': finished 2024-03-26T02:04:10.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi072-82404-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:10.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app2"}]: dispatch 2024-03-26T02:04:10.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1118612117' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi072-83605-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi072-83605-2"}]: dispatch 2024-03-26T02:04:10.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:10.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4017972454' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:04:10.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1381361081' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi072-82404-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T02:04:10.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/107770966' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-4", "tierpool": "test-rados-api-smithi072-82692-4-cache"}]: dispatch 2024-03-26T02:04:10.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T02:04:10.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3681383293' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:10.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: threexx 2024-03-26T02:04:10.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:10.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T02:04:10.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi072-82404-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:10.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:10.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi072-82387-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:10.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:10.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi072-83605-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:10.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:10.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi072-82938-1"}]': finished 2024-03-26T02:04:10.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi072-82404-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:10.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app2"}]: dispatch 2024-03-26T02:04:10.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1118612117' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi072-83605-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi072-83605-2"}]: dispatch 2024-03-26T02:04:10.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:10.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4017972454' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:04:10.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1381361081' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi072-82404-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T02:04:10.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/107770966' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-4", "tierpool": "test-rados-api-smithi072-82692-4-cache"}]: dispatch 2024-03-26T02:04:10.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T02:04:10.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3681383293' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:10.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:04:10.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi072-82496-10"}]: dispatch 2024-03-26T02:04:10.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi072-82404-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T02:04:10.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-4", "tierpool": "test-rados-api-smithi072-82692-4-cache"}]: dispatch 2024-03-26T02:04:10.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi072-83605-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi072-83605-2"}]: dispatch 2024-03-26T02:04:10.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:10.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:10.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/351269750' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:10.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:10.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi072-82938-1"}]: dispatch 2024-03-26T02:04:10.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi072-82496-10"}]: dispatch 2024-03-26T02:04:10.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi072-82404-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T02:04:10.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-4", "tierpool": "test-rados-api-smithi072-82692-4-cache"}]: dispatch 2024-03-26T02:04:10.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi072-83605-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi072-83605-2"}]: dispatch 2024-03-26T02:04:10.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:10.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:10.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/351269750' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:10.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:10.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:04:10.870 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: got: 2024-03-26T02:04:09.292701+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "t api_lock_pp: Running main() from gmock_main.cc 2024-03-26T02:04:10.870 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-03-26T02:04:10.870 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-03-26T02:04:10.870 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-03-26T02:04:10.870 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: seed 82496 2024-03-26T02:04:10.870 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-03-26T02:04:10.870 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (99 ms) 2024-03-26T02:04:10.870 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-03-26T02:04:10.870 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (8 ms) 2024-03-26T02:04:10.870 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-03-26T02:04:10.871 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1208 ms) 2024-03-26T02:04:10.871 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-03-26T02:04:10.871 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1086 ms) 2024-03-26T02:04:10.871 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-03-26T02:04:10.871 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (3 ms) 2024-03-26T02:04:10.871 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-03-26T02:04:10.871 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (4 ms) 2024-03-26T02:04:10.871 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-03-26T02:04:10.871 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (5 ms) 2024-03-26T02:04:10.871 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-03-26T02:04:10.871 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (3 ms) 2024-03-26T02:04:10.871 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2416 ms total) 2024-03-26T02:04:10.871 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: 2024-03-26T02:04:10.871 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-03-26T02:04:10.871 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-03-26T02:04:10.872 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1127 ms) 2024-03-26T02:04:10.872 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-03-26T02:04:10.872 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (13 ms) 2024-03-26T02:04:10.872 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-03-26T02:04:10.872 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1064 ms) 2024-03-26T02:04:10.872 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-03-26T02:04:10.872 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1003 ms) 2024-03-26T02:04:10.872 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-03-26T02:04:10.872 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (4 ms) 2024-03-26T02:04:10.872 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-03-26T02:04:10.872 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (3 ms) 2024-03-26T02:04:10.872 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-03-26T02:04:10.872 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (4 ms) 2024-03-26T02:04:10.872 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-03-26T02:04:10.873 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (3 ms) 2024-03-26T02:04:10.873 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3222 ms total) 2024-03-26T02:04:10.873 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: 2024-03-26T02:04:10.873 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-03-26T02:04:10.873 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (21792 ms total) 2024-03-26T02:04:10.873 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-03-26T02:04:11.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/351269750' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:11.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:11.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: pgmap v87: 812 pgs: 328 unknown, 32 creating+peering, 452 active+clean; 145 MiB data, 716 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:04:11.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: Health check update: 14 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:11.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:11.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi072-83563-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi072-83563-2"}]': finished 2024-03-26T02:04:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi072-82938-1"}]': finished 2024-03-26T02:04:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi072-82496-10"}]': finished 2024-03-26T02:04:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi072-82404-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-26T02:04:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-4", "tierpool": "test-rados-api-smithi072-82692-4-cache"}]': finished 2024-03-26T02:04:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/107770966' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-4", "overlaypool": "test-rados-api-smithi072-82692-4-cache"}]: dispatch 2024-03-26T02:04:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi072-82564-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-26T02:04:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T02:04:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T02:04:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/692566042' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/711058968' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4133273429' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:11.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/351269750' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:11.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:11.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: pgmap v87: 812 pgs: 328 unknown, 32 creating+peering, 452 active+clean; 145 MiB data, 716 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:04:11.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: Health check update: 14 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:11.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:11.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi072-83563-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi072-83563-2"}]': finished 2024-03-26T02:04:11.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi072-82938-1"}]': finished 2024-03-26T02:04:11.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi072-82496-10"}]': finished 2024-03-26T02:04:11.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi072-82404-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-26T02:04:11.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-4", "tierpool": "test-rados-api-smithi072-82692-4-cache"}]': finished 2024-03-26T02:04:11.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:11.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:11.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/107770966' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-4", "overlaypool": "test-rados-api-smithi072-82692-4-cache"}]: dispatch 2024-03-26T02:04:11.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi072-82564-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-26T02:04:11.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T02:04:11.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T02:04:11.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/692566042' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:11.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/711058968' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:11.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4133273429' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:11.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/351269750' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi072-82904-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:11.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3769470933' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:11.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi072-82496-10"}]: dispatch 2024-03-26T02:04:11.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-4", "overlaypool": "test-rados-api-smithi072-82692-4-cache"}]: dispatch 2024-03-26T02:04:11.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-26T02:04:11.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4045723459' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T02:04:11.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T02:04:11.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi072-82904-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:11.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:11.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:11.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:11.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi072-83605-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi072-83605-2"}]': finished 2024-03-26T02:04:11.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi072-82496-10"}]': finished 2024-03-26T02:04:11.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/107770966' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T02:04:11.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/351269750' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi072-82904-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:11.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3769470933' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:11.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi072-82496-10"}]: dispatch 2024-03-26T02:04:11.224 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-4", "overlaypool": "test-rados-api-smithi072-82692-4-cache"}]: dispatch 2024-03-26T02:04:11.224 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-26T02:04:11.224 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4045723459' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T02:04:11.224 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T02:04:11.224 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi072-82904-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:11.224 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:11.224 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:11.224 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:11.224 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi072-83605-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi072-83605-2"}]': finished 2024-03-26T02:04:11.224 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi072-82496-10"}]': finished 2024-03-26T02:04:11.224 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/107770966' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T02:04:11.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/351269750' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:11.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:11.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: pgmap v87: 812 pgs: 328 unknown, 32 creating+peering, 452 active+clean; 145 MiB data, 716 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:04:11.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: Health check update: 14 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:11.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:11.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi072-83563-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi072-83563-2"}]': finished 2024-03-26T02:04:11.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi072-82938-1"}]': finished 2024-03-26T02:04:11.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi072-82496-10"}]': finished 2024-03-26T02:04:11.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi072-82404-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-26T02:04:11.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-4", "tierpool": "test-rados-api-smithi072-82692-4-cache"}]': finished 2024-03-26T02:04:11.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi072-82564-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:11.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:11.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/107770966' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-4", "overlaypool": "test-rados-api-smithi072-82692-4-cache"}]: dispatch 2024-03-26T02:04:11.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi072-82564-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-26T02:04:11.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T02:04:11.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T02:04:11.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/692566042' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:11.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/711058968' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:11.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4133273429' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:11.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/351269750' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi072-82904-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:11.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3769470933' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:11.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi072-82496-10"}]: dispatch 2024-03-26T02:04:11.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-4", "overlaypool": "test-rados-api-smithi072-82692-4-cache"}]: dispatch 2024-03-26T02:04:11.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-26T02:04:11.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4045723459' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T02:04:11.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T02:04:11.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi072-82904-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:11.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:11.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:11.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:11.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi072-83605-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi072-83605-2"}]': finished 2024-03-26T02:04:11.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/231355520' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi072-82496-10"}]': finished 2024-03-26T02:04:11.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/107770966' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T02:04:11.575 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: Running main() from gmock_main.cc 2024-03-26T02:04:11.575 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-03-26T02:04:11.575 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [----------] Global test environment set-up. 2024-03-26T02:04:11.575 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-03-26T02:04:11.576 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-03-26T02:04:11.576 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [ OK ] LibRadosStat.Stat (330 ms) 2024-03-26T02:04:11.576 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-03-26T02:04:11.576 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (5 ms) 2024-03-26T02:04:11.576 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-03-26T02:04:11.576 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [ OK ] LibRadosStat.StatNS (15 ms) 2024-03-26T02:04:11.576 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-03-26T02:04:11.576 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (0 ms) 2024-03-26T02:04:11.576 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-03-26T02:04:11.576 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (4 ms) 2024-03-26T02:04:11.576 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [----------] 5 tests from LibRadosStat (354 ms total) 2024-03-26T02:04:11.576 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: 2024-03-26T02:04:11.576 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-03-26T02:04:11.576 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-03-26T02:04:11.576 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (2111 ms) 2024-03-26T02:04:11.577 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-03-26T02:04:11.577 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (31 ms) 2024-03-26T02:04:11.577 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-03-26T02:04:11.577 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (0 ms) 2024-03-26T02:04:11.577 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-03-26T02:04:11.577 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (2 ms) 2024-03-26T02:04:11.577 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (2144 ms total) 2024-03-26T02:04:11.577 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: 2024-03-26T02:04:11.577 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [----------] Global test environment tear-down 2024-03-26T02:04:11.577 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (22408 ms total) 2024-03-26T02:04:11.577 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat: [ PASSED ] 9 tests. 2024-03-26T02:04:11.853 INFO:tasks.workunit.client.0.smithi072.stdout: list: Running main() from gmock_main.cc 2024-03-26T02:04:11.853 INFO:tasks.workunit.client.0.smithi072.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-03-26T02:04:11.853 INFO:tasks.workunit.client.0.smithi072.stdout: list: [----------] Global test environment set-up. 2024-03-26T02:04:11.853 INFO:tasks.workunit.client.0.smithi072.stdout: list: [----------] 3 tests from NeoradosList 2024-03-26T02:04:11.853 INFO:tasks.workunit.client.0.smithi072.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-03-26T02:04:11.853 INFO:tasks.workunit.client.0.smithi072.stdout: list: [ OK ] NeoradosList.ListObjects (10917 ms) 2024-03-26T02:04:11.854 INFO:tasks.workunit.client.0.smithi072.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-03-26T02:04:11.854 INFO:tasks.workunit.client.0.smithi072.stdout: list: [ OK ] NeoradosList.ListObjectsNS (8442 ms) 2024-03-26T02:04:11.854 INFO:tasks.workunit.client.0.smithi072.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-03-26T02:04:11.854 INFO:tasks.workunit.client.0.smithi072.stdout: list: [ OK ] NeoradosList.ListObjectsMany (3059 ms) 2024-03-26T02:04:11.854 INFO:tasks.workunit.client.0.smithi072.stdout: list: [----------] 3 tests from NeoradosList (22418 ms total) 2024-03-26T02:04:11.854 INFO:tasks.workunit.client.0.smithi072.stdout: list: 2024-03-26T02:04:11.854 INFO:tasks.workunit.client.0.smithi072.stdout: list: [----------] Global test environment tear-down 2024-03-26T02:04:11.854 INFO:tasks.workunit.client.0.smithi072.stdout: list: [==========] 3 tests from 1 test suite ran. (22418 ms total) 2024-03-26T02:04:11.854 INFO:tasks.workunit.client.0.smithi072.stdout: list: [ PASSED ] 3 tests. 2024-03-26T02:04:12.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: fourxx 2024-03-26T02:04:12.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:04:12.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-4", "overlaypool": "test-rados-api-smithi072-82692-4-cache"}]': finished 2024-03-26T02:04:12.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T02:04:12.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi072-82904-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:12.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi072-82723-7"}]': finished 2024-03-26T02:04:12.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/351269750' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi072-82904-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:12.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T02:04:12.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3769470933' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:12.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3758764848' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:12.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T02:04:12.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:12.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi072-82904-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:12.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:12.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T02:04:12.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:12.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T02:04:12.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:12.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T02:04:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: fourxx 2024-03-26T02:04:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:04:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-4", "overlaypool": "test-rados-api-smithi072-82692-4-cache"}]': finished 2024-03-26T02:04:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T02:04:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi072-82904-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi072-82723-7"}]': finished 2024-03-26T02:04:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/351269750' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi072-82904-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T02:04:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3769470933' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3758764848' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T02:04:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:12.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi072-82904-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:12.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:12.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T02:04:12.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:12.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T02:04:12.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:12.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T02:04:12.472 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:12.472 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:12.473 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:04:12.473 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7"}]': finished 2024-03-26T02:04:12.473 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22"}]': finished 2024-03-26T02:04:12.473 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi072-82429-16"}]': finished 2024-03-26T02:04:12.473 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T02:04:12.473 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:12.473 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi072-82723-7"}]': finished 2024-03-26T02:04:12.473 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-4-cache", "mode": "writeback"}]': finished 2024-03-26T02:04:12.473 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:12.473 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/692566042' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:12.473 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4133273429' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:12.473 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/711058968' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:12.473 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T02:04:12.473 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2622638446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi072-82387-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:12.473 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/107770966' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-4"}]: dispatch 2024-03-26T02:04:12.474 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T02:04:12.474 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2332830424' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:12.474 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/964239072' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi072-83563-2"}]: dispatch 2024-03-26T02:04:12.474 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:12.474 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:12.474 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T02:04:12.474 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:12.474 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi072-82387-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:12.474 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2586181871' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:12.474 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:12.474 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi072-83563-2"}]: dispatch 2024-03-26T02:04:12.474 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:12.475 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:12.475 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:12.475 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:04:12.475 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7"}]': finished 2024-03-26T02:04:12.475 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22"}]': finished 2024-03-26T02:04:12.475 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi072-82429-16"}]': finished 2024-03-26T02:04:12.475 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T02:04:12.475 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:12.475 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi072-82723-7"}]': finished 2024-03-26T02:04:12.475 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-4-cache", "mode": "writeback"}]': finished 2024-03-26T02:04:12.475 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:12.475 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/692566042' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:12.476 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4133273429' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:12.476 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/711058968' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:12.476 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T02:04:12.476 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2622638446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi072-82387-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:12.476 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/107770966' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-4"}]: dispatch 2024-03-26T02:04:12.476 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T02:04:12.476 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2332830424' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:12.476 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/964239072' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi072-83563-2"}]: dispatch 2024-03-26T02:04:12.476 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:12.476 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:12.476 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T02:04:12.476 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:12.476 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi072-82387-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:12.476 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2586181871' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:12.476 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:12.477 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi072-83563-2"}]: dispatch 2024-03-26T02:04:12.477 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:12.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: fourxx 2024-03-26T02:04:12.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:04:12.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-4", "overlaypool": "test-rados-api-smithi072-82692-4-cache"}]': finished 2024-03-26T02:04:12.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T02:04:12.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi072-82904-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:12.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi072-82723-7"}]': finished 2024-03-26T02:04:12.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/351269750' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi072-82904-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:12.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T02:04:12.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3769470933' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:12.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3758764848' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:12.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T02:04:12.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:12.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi072-82904-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:12.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:12.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T02:04:12.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:12.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T02:04:12.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi072-82723-7"}]: dispatch 2024-03-26T02:04:12.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T02:04:12.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:12.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:12.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:04:12.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi072-82877-7"}]': finished 2024-03-26T02:04:12.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi072-82437-22"}]': finished 2024-03-26T02:04:12.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi072-82429-16"}]': finished 2024-03-26T02:04:12.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T02:04:12.566 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:12.566 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi072-82723-7"}]': finished 2024-03-26T02:04:12.566 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-4-cache", "mode": "writeback"}]': finished 2024-03-26T02:04:12.566 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:12.566 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/692566042' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:12.566 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4133273429' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:12.566 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/711058968' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:12.566 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/702115242' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T02:04:12.566 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2622638446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi072-82387-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:12.566 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/107770966' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-4"}]: dispatch 2024-03-26T02:04:12.566 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T02:04:12.567 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2332830424' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:12.567 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/964239072' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi072-83563-2"}]: dispatch 2024-03-26T02:04:12.567 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi072-82877-7"}]: dispatch 2024-03-26T02:04:12.567 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi072-82429-16"}]: dispatch 2024-03-26T02:04:12.567 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T02:04:12.567 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi072-82437-22"}]: dispatch 2024-03-26T02:04:12.567 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi072-82387-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:12.567 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2586181871' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:12.567 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:12.567 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi072-83563-2"}]: dispatch 2024-03-26T02:04:12.567 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:12.625 INFO:tasks.workunit.client.0.smithi072.stdout:ibRadosIoECPP.OverlappingWriteRoundTripPP 2024-03-26T02:04:12.625 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (6 ms) 2024-03-26T02:04:12.625 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-03-26T02:04:12.625 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (6 ms) 2024-03-26T02:04:12.625 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-03-26T02:04:12.626 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (4 ms) 2024-03-26T02:04:12.626 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-03-26T02:04:12.626 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (6 ms) 2024-03-26T02:04:12.626 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-03-26T02:04:12.626 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (4 ms) 2024-03-26T02:04:12.626 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-03-26T02:04:12.626 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (3 ms) 2024-03-26T02:04:12.626 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-03-26T02:04:12.626 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (3 ms) 2024-03-26T02:04:12.626 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-03-26T02:04:12.626 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (7 ms) 2024-03-26T02:04:12.626 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-03-26T02:04:12.626 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (4 ms) 2024-03-26T02:04:12.626 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-03-26T02:04:12.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (9 ms) 2024-03-26T02:04:12.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-03-26T02:04:12.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (5 ms) 2024-03-26T02:04:12.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-03-26T02:04:12.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (7 ms) 2024-03-26T02:04:12.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (1212 ms total) 2024-03-26T02:04:12.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: 2024-03-26T02:04:12.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [----------] Global test environment tear-down 2024-03-26T02:04:12.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (23567 ms total) 2024-03-26T02:04:12.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-03-26T02:04:12.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: Running main() from gmock_main.cc 2024-03-26T02:04:12.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-03-26T02:04:12.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [----------] Global test environment set-up. 2024-03-26T02:04:12.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-03-26T02:04:12.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-03-26T02:04:12.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (95 ms) 2024-03-26T02:04:12.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-03-26T02:04:12.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-03-26T02:04:12.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-03-26T02:04:12.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: no timeout :/ 2024-03-26T02:04:12.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: no timeout :/ 2024-03-26T02:04:12.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: no timeout :/ 2024-03-26T02:04:12.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: no timeout :/ 2024-03-26T02:04:12.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: no timeout :/ 2024-03-26T02:04:12.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (31 ms) 2024-03-26T02:04:12.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-03-26T02:04:12.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (6 ms) 2024-03-26T02:04:12.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-03-26T02:04:12.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ OK ] LibRadosIo.Checksum (4 ms) 2024-03-26T02:04:12.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-03-26T02:04:12.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (5 ms) 2024-03-26T02:04:12.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-03-26T02:04:12.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (2 ms) 2024-03-26T02:04:12.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-03-26T02:04:12.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (5 ms) 2024-03-26T02:04:12.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-03-26T02:04:12.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (2 ms) 2024-03-26T02:04:12.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-03-26T02:04:12.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ OK ] LibRadosIo.TruncTest (2 ms) 2024-03-26T02:04:12.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-03-26T02:04:12.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (3 ms) 2024-03-26T02:04:12.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-03-26T02:04:12.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (3 ms) 2024-03-26T02:04:12.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-03-26T02:04:12.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ OK ] LibRadosIo.RmXattr (7 ms) 2024-03-26T02:04:12.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-03-26T02:04:12.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ OK ] LibRadosIo.XattrIter (4 ms) 2024-03-26T02:04:12.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [----------] 14 tests from LibRadosIo (169 ms total) 2024-03-26T02:04:12.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: 2024-03-26T02:04:12.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-03-26T02:04:12.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-03-26T02:04:12.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1121 ms) 2024-03-26T02:04:12.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-03-26T02:04:12.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (6 ms) 2024-03-26T02:04:12.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-03-26T02:04:12.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (5 ms) 2024-03-26T02:04:12.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-03-26T02:04:12.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (6 ms) 2024-03-26T02:04:12.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-03-26T02:04:12.631 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (8 ms) 2024-03-26T02:04:12.631 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-03-26T02:04:12.631 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (5 ms) 2024-03-26T02:04:12.631 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-03-26T02:04:12.631 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (2 ms) 2024-03-26T02:04:12.631 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-03-26T02:04:12.631 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (3 ms) 2024-03-26T02:04:12.631 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-03-26T02:04:12.631 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (9 ms) 2024-03-26T02:04:12.631 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-03-26T02:04:12.631 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (4 ms) 2024-03-26T02:04:12.631 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1169 ms total) 2024-03-26T02:04:12.631 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: 2024-03-26T02:04:12.631 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [----------] Global test environment tear-down 2024-03-26T02:04:12.631 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [==========] 24 tests from 2 test suites ran. (23567 ms total) 2024-03-26T02:04:12.632 INFO:tasks.workunit.client.0.smithi072.stdout: api_io: [ PASSED ] 24 tests. 2024-03-26T02:04:12.643 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-03-26T02:04:12.643 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-03-26T02:04:12.643 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-03-26T02:04:12.643 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-03-26T02:04:12.643 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: seed 82877 2024-03-26T02:04:12.643 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-03-26T02:04:12.643 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (272 ms) 2024-03-26T02:04:12.643 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-03-26T02:04:12.643 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (41 ms) 2024-03-26T02:04:12.643 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-03-26T02:04:12.643 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (1 ms) 2024-03-26T02:04:12.644 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-03-26T02:04:12.644 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (5 ms) 2024-03-26T02:04:12.644 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-03-26T02:04:12.644 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (13 ms) 2024-03-26T02:04:12.644 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (332 ms total) 2024-03-26T02:04:12.644 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: 2024-03-26T02:04:12.644 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-03-26T02:04:12.644 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-03-26T02:04:12.644 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1108 ms) 2024-03-26T02:04:12.644 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-03-26T02:04:12.644 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (0 ms) 2024-03-26T02:04:12.644 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-03-26T02:04:12.644 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (8 ms) 2024-03-26T02:04:12.644 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-03-26T02:04:12.644 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (14 ms) 2024-03-26T02:04:12.644 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1130 ms total) 2024-03-26T02:04:12.645 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: 2024-03-26T02:04:12.645 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-03-26T02:04:12.645 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (23419 ms total) 2024-03-26T02:04:12.645 INFO:tasks.workunit.client.0.smithi072.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-03-26T02:04:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:13 smithi173 ceph-mon[33839]: pgmap v90: 828 pgs: 104 creating+peering, 104 unknown, 620 active+clean; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-26T02:04:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:13 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-4"}]: dispatch 2024-03-26T02:04:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:13 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi072-82938-6", "pg_num": 4}]: dispatch 2024-03-26T02:04:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:13 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-26T02:04:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:13 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:13 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi072-82904-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi072-82904-12"}]': finished 2024-03-26T02:04:13.654 INFO:tasks.workunit.client.0.smithi072.stdout: ec_io: Running main() from gmock_main.cc 2024-03-26T02:04:13.654 INFO:tasks.workunit.client.0.smithi072.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-03-26T02:04:13.654 INFO:tasks.workunit.client.0.smithi072.stdout: ec_io: [----------] Global test environment set-up. 2024-03-26T02:04:13.654 INFO:tasks.workunit.client.0.smithi072.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-03-26T02:04:13.654 INFO:tasks.workunit.client.0.smithi072.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-03-26T02:04:13.654 INFO:tasks.workunit.client.0.smithi072.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (17066 ms) 2024-03-26T02:04:13.654 INFO:tasks.workunit.client.0.smithi072.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-03-26T02:04:13.654 INFO:tasks.workunit.client.0.smithi072.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (7150 ms) 2024-03-26T02:04:13.655 INFO:tasks.workunit.client.0.smithi072.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (24216 ms total) 2024-03-26T02:04:13.655 INFO:tasks.workunit.client.0.smithi072.stdout: ec_io: 2024-03-26T02:04:13.655 INFO:tasks.workunit.client.0.smithi072.stdout: ec_io: [----------] Global test environment tear-down 2024-03-26T02:04:13.655 INFO:tasks.workunit.client.0.smithi072.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (24216 ms total) 2024-03-26T02:04:13.655 INFO:tasks.workunit.client.0.smithi072.stdout: ec_io: [ PASSED ] 2 tests. 2024-03-26T02:04:13.670 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: Running main() from gmock_main.cc 2024-03-26T02:04:13.670 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-03-26T02:04:13.670 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [----------] Global test environment set-up. 2024-03-26T02:04:13.671 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-03-26T02:04:13.671 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-03-26T02:04:13.671 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (311 ms) 2024-03-26T02:04:13.671 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-03-26T02:04:13.671 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ OK ] LibRadosLock.LockShared (10 ms) 2024-03-26T02:04:13.671 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-03-26T02:04:13.671 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1122 ms) 2024-03-26T02:04:13.671 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-03-26T02:04:13.671 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1039 ms) 2024-03-26T02:04:13.671 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-03-26T02:04:13.671 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (4 ms) 2024-03-26T02:04:13.671 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-03-26T02:04:13.672 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ OK ] LibRadosLock.Unlock (3 ms) 2024-03-26T02:04:13.672 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-03-26T02:04:13.672 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (4 ms) 2024-03-26T02:04:13.672 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-03-26T02:04:13.672 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (3 ms) 2024-03-26T02:04:13.672 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [----------] 8 tests from LibRadosLock (2496 ms total) 2024-03-26T02:04:13.672 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: 2024-03-26T02:04:13.672 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-03-26T02:04:13.672 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-03-26T02:04:13.672 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (866 ms) 2024-03-26T02:04:13.672 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-03-26T02:04:13.672 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (4 ms) 2024-03-26T02:04:13.672 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-03-26T02:04:13.673 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1008 ms) 2024-03-26T02:04:13.673 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-03-26T02:04:13.673 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1009 ms) 2024-03-26T02:04:13.673 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-03-26T02:04:13.673 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (6 ms) 2024-03-26T02:04:13.673 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-03-26T02:04:13.673 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (5 ms) 2024-03-26T02:04:13.673 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-03-26T02:04:13.673 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (5 ms) 2024-03-26T02:04:13.673 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-03-26T02:04:13.673 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (3 ms) 2024-03-26T02:04:13.673 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (2906 ms total) 2024-03-26T02:04:13.673 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: 2024-03-26T02:04:13.674 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [----------] Global test environment tear-down 2024-03-26T02:04:13.674 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (24529 ms total) 2024-03-26T02:04:13.674 INFO:tasks.workunit.client.0.smithi072.stdout: api_lock: [ PASSED ] 16 tests. 2024-03-26T02:04:13.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:13 smithi072 ceph-mon[30177]: pgmap v90: 828 pgs: 104 creating+peering, 104 unknown, 620 active+clean; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-26T02:04:13.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:13 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-4"}]: dispatch 2024-03-26T02:04:13.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:13 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi072-82938-6", "pg_num": 4}]: dispatch 2024-03-26T02:04:13.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:13 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-26T02:04:13.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:13 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:13.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:13 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi072-82904-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi072-82904-12"}]': finished 2024-03-26T02:04:13.727 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:13 smithi072 ceph-mon[41619]: pgmap v90: 828 pgs: 104 creating+peering, 104 unknown, 620 active+clean; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-03-26T02:04:13.727 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:13 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-4"}]: dispatch 2024-03-26T02:04:13.727 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:13 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi072-82938-6", "pg_num": 4}]: dispatch 2024-03-26T02:04:13.727 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:13 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-26T02:04:13.728 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:13 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:13.728 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:13 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi072-82904-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi072-82904-12"}]': finished 2024-03-26T02:04:15.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi072-82877-7"}]': finished 2024-03-26T02:04:15.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi072-82429-16"}]': finished 2024-03-26T02:04:15.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key1"}]': finished 2024-03-26T02:04:15.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi072-82437-22"}]': finished 2024-03-26T02:04:15.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi072-82387-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:15.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:15.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi072-83563-2"}]': finished 2024-03-26T02:04:15.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi072-82604-10"}]': finished 2024-03-26T02:04:15.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-4"}]': finished 2024-03-26T02:04:15.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi072-82938-6", "pg_num": 4}]': finished 2024-03-26T02:04:15.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/964239072' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi072-83563-2"}]: dispatch 2024-03-26T02:04:15.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2586181871' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:15.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1118612117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi072-83605-2"}]: dispatch 2024-03-26T02:04:15.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/107770966' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-4", "tierpool": "test-rados-api-smithi072-82692-4-cache"}]: dispatch 2024-03-26T02:04:15.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T02:04:15.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4", "tierpool": "test-rados-api-smithi072-82938-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:15.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi072-83563-2"}]: dispatch 2024-03-26T02:04:15.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-4", "tierpool": "test-rados-api-smithi072-82692-4-cache"}]: dispatch 2024-03-26T02:04:15.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:15.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi072-83605-2"}]: dispatch 2024-03-26T02:04:15.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:15.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:04:15.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4", "tierpool": "test-rados-api-smithi072-82938-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:04:15.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi072-83563-2"}]': finished 2024-03-26T02:04:15.316 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-4", "tierpool": "test-rados-api-smithi072-82692-4-cache"}]': finished 2024-03-26T02:04:15.316 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi072-82604-10"}]': finished 2024-03-26T02:04:15.316 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi072-83605-2"}]': finished 2024-03-26T02:04:15.316 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T02:04:15.316 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/966957026' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:15.316 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1118612117' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi072-83605-2"}]: dispatch 2024-03-26T02:04:15.316 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4", "overlaypool": "test-rados-api-smithi072-82938-6"}]: dispatch 2024-03-26T02:04:15.316 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi072-83605-2"}]: dispatch 2024-03-26T02:04:15.316 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:15.348 INFO:tasks.workunit.client.0.smithi072.stdout:estprofile-LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:15.348 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (13489 ms) 2024-03-26T02:04:15.348 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (24881 ms total) 2024-03-26T02:04:15.348 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: 2024-03-26T02:04:15.348 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: [----------] Global test environment tear-down 2024-03-26T02:04:15.348 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (24882 ms total) 2024-03-26T02:04:15.349 INFO:tasks.workunit.client.0.smithi072.stdout: api_cmd: [ PASSED ] 4 tests. 2024-03-26T02:04:15.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi072-82877-7"}]': finished 2024-03-26T02:04:15.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi072-82429-16"}]': finished 2024-03-26T02:04:15.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key1"}]': finished 2024-03-26T02:04:15.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi072-82437-22"}]': finished 2024-03-26T02:04:15.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi072-82387-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:15.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:15.473 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi072-83563-2"}]': finished 2024-03-26T02:04:15.474 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi072-82604-10"}]': finished 2024-03-26T02:04:15.474 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-4"}]': finished 2024-03-26T02:04:15.474 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi072-82938-6", "pg_num": 4}]': finished 2024-03-26T02:04:15.474 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/964239072' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi072-83563-2"}]: dispatch 2024-03-26T02:04:15.474 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2586181871' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:15.474 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1118612117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi072-83605-2"}]: dispatch 2024-03-26T02:04:15.474 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/107770966' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-4", "tierpool": "test-rados-api-smithi072-82692-4-cache"}]: dispatch 2024-03-26T02:04:15.474 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T02:04:15.474 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4", "tierpool": "test-rados-api-smithi072-82938-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:15.474 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi072-83563-2"}]: dispatch 2024-03-26T02:04:15.474 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-4", "tierpool": "test-rados-api-smithi072-82692-4-cache"}]: dispatch 2024-03-26T02:04:15.474 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:15.474 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi072-83605-2"}]: dispatch 2024-03-26T02:04:15.474 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:15.475 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:04:15.475 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4", "tierpool": "test-rados-api-smithi072-82938-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:04:15.475 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi072-83563-2"}]': finished 2024-03-26T02:04:15.475 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-4", "tierpool": "test-rados-api-smithi072-82692-4-cache"}]': finished 2024-03-26T02:04:15.475 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi072-82604-10"}]': finished 2024-03-26T02:04:15.475 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi072-83605-2"}]': finished 2024-03-26T02:04:15.475 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T02:04:15.475 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/966957026' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:15.475 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1118612117' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi072-83605-2"}]: dispatch 2024-03-26T02:04:15.475 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4", "overlaypool": "test-rados-api-smithi072-82938-6"}]: dispatch 2024-03-26T02:04:15.475 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi072-83605-2"}]: dispatch 2024-03-26T02:04:15.475 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:15.476 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:04:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:04:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:04:15.476 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi072-82877-7"}]': finished 2024-03-26T02:04:15.476 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi072-82429-16"}]': finished 2024-03-26T02:04:15.476 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi072-82564-1","app":"app1","key":"key1"}]': finished 2024-03-26T02:04:15.476 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi072-82437-22"}]': finished 2024-03-26T02:04:15.476 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi072-82387-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:15.476 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:15.476 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi072-83563-2"}]': finished 2024-03-26T02:04:15.476 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi072-82604-10"}]': finished 2024-03-26T02:04:15.476 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-4"}]': finished 2024-03-26T02:04:15.477 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi072-82938-6", "pg_num": 4}]': finished 2024-03-26T02:04:15.477 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/964239072' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi072-83563-2"}]: dispatch 2024-03-26T02:04:15.477 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2586181871' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:15.477 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1118612117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi072-83605-2"}]: dispatch 2024-03-26T02:04:15.477 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/107770966' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-4", "tierpool": "test-rados-api-smithi072-82692-4-cache"}]: dispatch 2024-03-26T02:04:15.477 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T02:04:15.477 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4", "tierpool": "test-rados-api-smithi072-82938-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:15.477 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi072-83563-2"}]: dispatch 2024-03-26T02:04:15.477 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-4", "tierpool": "test-rados-api-smithi072-82692-4-cache"}]: dispatch 2024-03-26T02:04:15.477 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi072-82604-10"}]: dispatch 2024-03-26T02:04:15.477 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi072-83605-2"}]: dispatch 2024-03-26T02:04:15.477 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:15.477 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:04:15.477 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4", "tierpool": "test-rados-api-smithi072-82938-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:04:15.478 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi072-83563-2"}]': finished 2024-03-26T02:04:15.478 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-4", "tierpool": "test-rados-api-smithi072-82692-4-cache"}]': finished 2024-03-26T02:04:15.478 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi072-82604-10"}]': finished 2024-03-26T02:04:15.478 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi072-83605-2"}]': finished 2024-03-26T02:04:15.478 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T02:04:15.478 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/966957026' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:15.478 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1118612117' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi072-83605-2"}]: dispatch 2024-03-26T02:04:15.478 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4", "overlaypool": "test-rados-api-smithi072-82938-6"}]: dispatch 2024-03-26T02:04:15.478 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi072-83605-2"}]: dispatch 2024-03-26T02:04:15.478 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:16.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:16 smithi072 ceph-mon[30177]: pgmap v93: 784 pgs: 204 unknown, 580 active+clean; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 3.5 KiB/s wr, 4 op/s 2024-03-26T02:04:16.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:16 smithi072 ceph-mon[30177]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:16.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:16 smithi072 ceph-mon[30177]: pool 'PoolQuotaPP_smithi072-82404-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-26T02:04:16.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:16 smithi072 ceph-mon[30177]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T02:04:16.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:16 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:16.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:16 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4", "overlaypool": "test-rados-api-smithi072-82938-6"}]': finished 2024-03-26T02:04:16.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:16 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi072-83605-2"}]': finished 2024-03-26T02:04:16.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:16 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:16.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:16 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4261530815' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi072-83605-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:16.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:16 smithi072 ceph-mon[30177]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T02:04:16.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:16 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82938-6", "mode": "writeback"}]: dispatch 2024-03-26T02:04:16.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:16 smithi072 ceph-mon[41619]: pgmap v93: 784 pgs: 204 unknown, 580 active+clean; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 3.5 KiB/s wr, 4 op/s 2024-03-26T02:04:16.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:16 smithi072 ceph-mon[41619]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:16.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:16 smithi072 ceph-mon[41619]: pool 'PoolQuotaPP_smithi072-82404-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-26T02:04:16.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:16 smithi072 ceph-mon[41619]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T02:04:16.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:16 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:16.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:16 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4", "overlaypool": "test-rados-api-smithi072-82938-6"}]': finished 2024-03-26T02:04:16.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:16 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi072-83605-2"}]': finished 2024-03-26T02:04:16.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:16 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:16.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:16 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4261530815' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi072-83605-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:16.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:16 smithi072 ceph-mon[41619]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T02:04:16.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:16 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82938-6", "mode": "writeback"}]: dispatch 2024-03-26T02:04:16.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:16 smithi173 ceph-mon[33839]: pgmap v93: 784 pgs: 204 unknown, 580 active+clean; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 3.5 KiB/s wr, 4 op/s 2024-03-26T02:04:16.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:16 smithi173 ceph-mon[33839]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:16.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:16 smithi173 ceph-mon[33839]: pool 'PoolQuotaPP_smithi072-82404-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-26T02:04:16.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:16 smithi173 ceph-mon[33839]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T02:04:16.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:16 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:16.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:16 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4", "overlaypool": "test-rados-api-smithi072-82938-6"}]': finished 2024-03-26T02:04:16.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:16 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi072-83605-2"}]': finished 2024-03-26T02:04:16.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:16 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:16.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:16 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4261530815' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi072-83605-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:16.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:16 smithi173 ceph-mon[33839]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T02:04:16.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:16 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82938-6", "mode": "writeback"}]: dispatch 2024-03-26T02:04:17.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:17 smithi173 ceph-mon[33839]: pgmap v96: 688 pgs: 18 creating+peering, 14 unknown, 656 active+clean; 145 MiB data, 781 MiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 4.7 KiB/s wr, 4 op/s 2024-03-26T02:04:17.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:17 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:17.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:17 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:04:17.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:17 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi072-83605-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:17.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:17 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82938-6", "mode": "writeback"}]': finished 2024-03-26T02:04:17.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:17 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/351269750' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:17.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:17 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2369745447' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi072-82387-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:17.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:17 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3103104122' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:17.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:17 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/116937037' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi072-82705-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:17.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:17 smithi173 ceph-mon[33839]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T02:04:17.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:17 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:17.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:17 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi072-82387-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:17.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:17 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi072-82705-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:17.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:17 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:17.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:17 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:17.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[41619]: pgmap v96: 688 pgs: 18 creating+peering, 14 unknown, 656 active+clean; 145 MiB data, 781 MiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 4.7 KiB/s wr, 4 op/s 2024-03-26T02:04:17.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:17.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:04:17.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi072-83605-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:17.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82938-6", "mode": "writeback"}]': finished 2024-03-26T02:04:17.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/351269750' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:17.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2369745447' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi072-82387-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:17.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3103104122' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:17.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/116937037' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi072-82705-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:17.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[41619]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T02:04:17.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:17.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi072-82387-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:17.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi072-82705-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:17.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:17.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:17.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[30177]: pgmap v96: 688 pgs: 18 creating+peering, 14 unknown, 656 active+clean; 145 MiB data, 781 MiB used, 714 GiB / 715 GiB avail; 3.7 KiB/s rd, 4.7 KiB/s wr, 4 op/s 2024-03-26T02:04:17.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:17.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:04:17.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi072-83605-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:17.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82938-6", "mode": "writeback"}]': finished 2024-03-26T02:04:17.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/351269750' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:17.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2369745447' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi072-82387-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:17.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3103104122' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:17.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/116937037' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi072-82705-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:17.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[30177]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T02:04:17.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:17.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi072-82387-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:17.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi072-82705-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:17.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:17.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:17 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:18.683 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-03-26T02:04:18.683 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-03-26T02:04:18.683 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-03-26T02:04:18.683 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-03-26T02:04:18.683 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-03-26T02:04:18.683 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: watch_notify_test_cb 2024-03-26T02:04:18.683 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (331 ms) 2024-03-26T02:04:18.683 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-03-26T02:04:18.683 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94090179000864 err -107 2024-03-26T02:04:18.683 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-26T02:04:18.684 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (17 ms) 2024-03-26T02:04:18.684 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-03-26T02:04:18.684 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94090179002240 err -107 2024-03-26T02:04:18.684 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-26T02:04:18.684 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (7 ms) 2024-03-26T02:04:18.684 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-03-26T02:04:18.684 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: watch_notify2_test_cb from 24640 notify_id 326417514496 cookie 94090179019104 2024-03-26T02:04:18.684 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (8 ms) 2024-03-26T02:04:18.684 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-03-26T02:04:18.684 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: watch_notify2_test_cb from 24640 notify_id 326417514496 cookie 94090179038336 2024-03-26T02:04:18.684 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (8 ms) 2024-03-26T02:04:18.684 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-03-26T02:04:18.684 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: watch_notify2_test_cb from 24640 notify_id 326417514497 cookie 94090179038240 2024-03-26T02:04:18.684 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (7 ms) 2024-03-26T02:04:18.684 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-03-26T02:04:18.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: watch_notify2_test_cb from 24640 notify_id 326417514498 cookie 94090179038240 2024-03-26T02:04:18.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: watch_notify2_test_cb from 24640 notify_id 326417514498 cookie 94090179061776 2024-03-26T02:04:18.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (10 ms) 2024-03-26T02:04:18.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-03-26T02:04:18.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: watch_notify2_test_cb from 24640 notify_id 326417514497 cookie 94090179038240 2024-03-26T02:04:18.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: watch_notify2_test_cb from 24640 notify_id 330712481794 cookie 94090179038240 2024-03-26T02:04:18.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3082 ms) 2024-03-26T02:04:18.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-03-26T02:04:18.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-03-26T02:04:18.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94090179038240 err -107 2024-03-26T02:04:18.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: watch_notify2_test_cb from 24640 notify_id 356482285573 cookie 94090179038240 2024-03-26T02:04:18.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5014 ms) 2024-03-26T02:04:18.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-03-26T02:04:18.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-03-26T02:04:18.686 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94090179038240 err -107 2024-03-26T02:04:18.686 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1004 ms) 2024-03-26T02:04:18.686 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (9488 ms total) 2024-03-26T02:04:18.686 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: 2024-03-26T02:04:18.686 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-03-26T02:04:18.686 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-03-26T02:04:18.686 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: watch_notify_test_cb 2024-03-26T02:04:18.686 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1175 ms) 2024-03-26T02:04:18.686 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1175 ms total) 2024-03-26T02:04:18.686 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: 2024-03-26T02:04:18.686 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-03-26T02:04:18.686 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (29461 ms total) 2024-03-26T02:04:18.686 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-03-26T02:04:18.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:18 smithi072 ceph-mon[30177]: pgmap v98: 648 pgs: 18 creating+peering, 110 unknown, 520 active+clean; 145 MiB data, 781 MiB used, 714 GiB / 715 GiB avail; 839 B/s rd, 1 op/s 2024-03-26T02:04:18.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi072-83605-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:18.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi072-82904-12"}]': finished 2024-03-26T02:04:18.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi072-82387-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:18.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi072-82705-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:18.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:18.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4261530815' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi072-83605-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi072-83605-3"}]: dispatch 2024-03-26T02:04:18.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/351269750' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:18.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:18 smithi072 ceph-mon[30177]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T02:04:18.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi072-83605-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi072-83605-3"}]: dispatch 2024-03-26T02:04:18.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:18.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4128383562' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi072-82728-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:18.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:18.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:18 smithi072 ceph-mon[41619]: pgmap v98: 648 pgs: 18 creating+peering, 110 unknown, 520 active+clean; 145 MiB data, 781 MiB used, 714 GiB / 715 GiB avail; 839 B/s rd, 1 op/s 2024-03-26T02:04:18.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi072-83605-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:18.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi072-82904-12"}]': finished 2024-03-26T02:04:18.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi072-82387-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:18.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi072-82705-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:18.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:18.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4261530815' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi072-83605-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi072-83605-3"}]: dispatch 2024-03-26T02:04:18.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/351269750' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:18.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:18 smithi072 ceph-mon[41619]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T02:04:18.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi072-83605-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi072-83605-3"}]: dispatch 2024-03-26T02:04:18.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:18.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4128383562' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi072-82728-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:18.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:19.062 INFO:tasks.workunit.client.0.smithi072.stdout: pool: Running main() from gmock_main.cc 2024-03-26T02:04:19.062 INFO:tasks.workunit.client.0.smithi072.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-03-26T02:04:19.062 INFO:tasks.workunit.client.0.smithi072.stdout: pool: [----------] Global test environment set-up. 2024-03-26T02:04:19.062 INFO:tasks.workunit.client.0.smithi072.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-03-26T02:04:19.062 INFO:tasks.workunit.client.0.smithi072.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-03-26T02:04:19.062 INFO:tasks.workunit.client.0.smithi072.stdout: pool: [ OK ] NeoRadosPools.PoolList (9980 ms) 2024-03-26T02:04:19.062 INFO:tasks.workunit.client.0.smithi072.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-03-26T02:04:19.062 INFO:tasks.workunit.client.0.smithi072.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (5020 ms) 2024-03-26T02:04:19.062 INFO:tasks.workunit.client.0.smithi072.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-03-26T02:04:19.062 INFO:tasks.workunit.client.0.smithi072.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (3023 ms) 2024-03-26T02:04:19.063 INFO:tasks.workunit.client.0.smithi072.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-03-26T02:04:19.063 INFO:tasks.workunit.client.0.smithi072.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (4360 ms) 2024-03-26T02:04:19.063 INFO:tasks.workunit.client.0.smithi072.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-03-26T02:04:19.063 INFO:tasks.workunit.client.0.smithi072.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (3925 ms) 2024-03-26T02:04:19.063 INFO:tasks.workunit.client.0.smithi072.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-03-26T02:04:19.063 INFO:tasks.workunit.client.0.smithi072.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (3294 ms) 2024-03-26T02:04:19.063 INFO:tasks.workunit.client.0.smithi072.stdout: pool: [----------] 6 tests from NeoRadosPools (29603 ms total) 2024-03-26T02:04:19.063 INFO:tasks.workunit.client.0.smithi072.stdout: pool: 2024-03-26T02:04:19.063 INFO:tasks.workunit.client.0.smithi072.stdout: pool: [----------] Global test environment tear-down 2024-03-26T02:04:19.063 INFO:tasks.workunit.client.0.smithi072.stdout: pool: [==========] 6 tests from 1 test suite ran. (29603 ms total) 2024-03-26T02:04:19.063 INFO:tasks.workunit.client.0.smithi072.stdout: pool: [ PASSED ] 6 tests. 2024-03-26T02:04:19.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:18 smithi173 ceph-mon[33839]: pgmap v98: 648 pgs: 18 creating+peering, 110 unknown, 520 active+clean; 145 MiB data, 781 MiB used, 714 GiB / 715 GiB avail; 839 B/s rd, 1 op/s 2024-03-26T02:04:19.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi072-83605-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:19.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi072-82904-12"}]': finished 2024-03-26T02:04:19.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi072-82387-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:19.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi072-82705-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:19.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:19.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4261530815' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi072-83605-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi072-83605-3"}]: dispatch 2024-03-26T02:04:19.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/351269750' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:19.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:18 smithi173 ceph-mon[33839]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T02:04:19.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi072-83605-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi072-83605-3"}]: dispatch 2024-03-26T02:04:19.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi072-82904-12"}]: dispatch 2024-03-26T02:04:19.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4128383562' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi072-82728-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:19.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:19.909 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-03-26T02:04:19.909 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-03-26T02:04:19.909 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-03-26T02:04:19.909 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-03-26T02:04:19.909 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-03-26T02:04:19.909 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: notify 2024-03-26T02:04:19.909 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1158 ms) 2024-03-26T02:04:19.909 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-03-26T02:04:19.909 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (2239 ms) 2024-03-26T02:04:19.910 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (3397 ms total) 2024-03-26T02:04:19.910 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: 2024-03-26T02:04:19.910 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-03-26T02:04:19.910 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-03-26T02:04:19.910 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: notify 2024-03-26T02:04:19.910 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (11 ms) 2024-03-26T02:04:19.910 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-03-26T02:04:19.910 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: notify 2024-03-26T02:04:19.910 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (4986 ms) 2024-03-26T02:04:19.910 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-03-26T02:04:19.910 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (5 ms) 2024-03-26T02:04:19.910 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-03-26T02:04:19.910 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (6 ms) 2024-03-26T02:04:19.910 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-03-26T02:04:19.911 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: handle_notify cookie 94113618202688 notify_id 395136991234 notifier_gid 15249 2024-03-26T02:04:19.911 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (8 ms) 2024-03-26T02:04:19.911 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-03-26T02:04:19.911 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: handle_notify cookie 94113618205600 notify_id 395136991232 notifier_gid 15249 2024-03-26T02:04:19.911 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (5 ms) 2024-03-26T02:04:19.911 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-03-26T02:04:19.911 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: handle_notify cookie 94113618205600 notify_id 395136991233 notifier_gid 15249 2024-03-26T02:04:19.911 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (10 ms) 2024-03-26T02:04:19.911 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-03-26T02:04:19.911 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: handle_notify cookie 94113618205600 notify_id 395136991234 notifier_gid 15249 2024-03-26T02:04:19.911 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (7 ms) 2024-03-26T02:04:19.911 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-03-26T02:04:19.911 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: handle_notify cookie 94113618205600 notify_id 395136991235 notifier_gid 15249 2024-03-26T02:04:19.911 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (10 ms) 2024-03-26T02:04:19.912 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-03-26T02:04:19.912 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: handle_notify cookie 94113618209456 notify_id 395136991236 notifier_gid 15249 2024-03-26T02:04:19.912 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (8 ms) 2024-03-26T02:04:19.912 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-03-26T02:04:19.912 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: trying... 2024-03-26T02:04:19.912 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: handle_notify cookie 94113618205600 notify_id 395136991235 notifier_gid 15249 2024-03-26T02:04:19.912 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: timed out 2024-03-26T02:04:19.912 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: flushing 2024-03-26T02:04:20.473 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:20 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi072-82904-12"}]': finished 2024-03-26T02:04:20.473 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4128383562' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi072-82728-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:20.473 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:20.473 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:20 smithi072 ceph-mon[30177]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T02:04:20.473 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:20.473 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:20 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:20.473 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:20.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:20 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi072-82904-12"}]': finished 2024-03-26T02:04:20.473 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4128383562' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi072-82728-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:20.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:20.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:20 smithi072 ceph-mon[41619]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T02:04:20.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:20.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:20 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:20.474 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:20.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:20 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi072-82904-12"}]': finished 2024-03-26T02:04:20.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4128383562' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi072-82728-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:20.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:20.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:20 smithi173 ceph-mon[33839]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T02:04:20.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:20.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:20 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:20.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:21.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:04:20 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:04:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:21 smithi173 ceph-mon[33839]: pgmap v101: 680 pgs: 224 unknown, 456 active+clean; 145 MiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 4.7 KiB/s wr, 1 op/s 2024-03-26T02:04:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:21 smithi173 ceph-mon[33839]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:21 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi072-83605-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi072-83605-3"}]': finished 2024-03-26T02:04:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:21 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:21 smithi173 ceph-mon[33839]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T02:04:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4268307674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1278673961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi072-82387-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:21.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:21 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi072-82387-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:21.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:21 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:21.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/454858173' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:21.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:21 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:21.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:21 smithi072 ceph-mon[41619]: pgmap v101: 680 pgs: 224 unknown, 456 active+clean; 145 MiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 4.7 KiB/s wr, 1 op/s 2024-03-26T02:04:21.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:21 smithi072 ceph-mon[41619]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:21.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:21 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi072-83605-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi072-83605-3"}]': finished 2024-03-26T02:04:21.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:21 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:21.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:21 smithi072 ceph-mon[41619]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T02:04:21.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4268307674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:21.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1278673961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi072-82387-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:21.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:21 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi072-82387-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:21.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:21 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:21.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/454858173' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:21.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:21 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:21.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:21 smithi072 ceph-mon[30177]: pgmap v101: 680 pgs: 224 unknown, 456 active+clean; 145 MiB data, 781 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 4.7 KiB/s wr, 1 op/s 2024-03-26T02:04:21.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:21 smithi072 ceph-mon[30177]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:21.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:21 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi072-83605-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi072-83605-3"}]': finished 2024-03-26T02:04:21.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:21 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:21.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:21 smithi072 ceph-mon[30177]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T02:04:21.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4268307674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:21.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1278673961' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi072-82387-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:21.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:21 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi072-82387-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:21.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:21 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:21.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/454858173' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:21.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:21 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:21.991 INFO:tasks.workunit.client.0.smithi072.stdout: api_ndry: 1 daemon active (1 hosts) 2024-03-26T02:04:21.991 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-26T02:04:21.991 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 2024-03-26T02:04:21.991 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: data: 2024-03-26T02:04:21.992 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: pools: 36 pools, 1092 pgs 2024-03-26T02:04:21.992 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: objects: 230 objects, 582 KiB 2024-03-26T02:04:21.992 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: usage: 235 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:04:21.992 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: pgs: 53.388% pgs unknown 2024-03-26T02:04:21.992 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 34.524% pgs not active 2024-03-26T02:04:21.992 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 583 unknown 2024-03-26T02:04:21.992 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 377 creating+peering 2024-03-26T02:04:21.992 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 132 active+clean 2024-03-26T02:04:21.992 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 2024-03-26T02:04:21.992 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: io: 2024-03-26T02:04:21.992 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: client: 852 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-26T02:04:21.992 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 2024-03-26T02:04:21.992 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 2024-03-26T02:04:21.993 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: [ OK ] LibRadosService.StatusFormat (7575 ms) 2024-03-26T02:04:21.993 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: [ RUN ] LibRadosService.Status 2024-03-26T02:04:21.993 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: [ OK ] LibRadosService.Status (20040 ms) 2024-03-26T02:04:21.993 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: [----------] 4 tests from LibRadosService (32705 ms total) 2024-03-26T02:04:21.993 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: 2024-03-26T02:04:21.993 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: [----------] Global test environment tear-down 2024-03-26T02:04:21.993 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: [==========] 4 tests from 1 test suite ran. (32705 ms total) 2024-03-26T02:04:21.993 INFO:tasks.workunit.client.0.smithi072.stdout: api_service: [ PASSED ] 4 tests. 2024-03-26T02:04:23.219 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-03-26T02:04:23.219 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: [----------] Global test environment set-up. 2024-03-26T02:04:23.220 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-03-26T02:04:23.220 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-03-26T02:04:23.220 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5061 ms) 2024-03-26T02:04:23.220 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-03-26T02:04:23.220 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (104 ms) 2024-03-26T02:04:23.220 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-03-26T02:04:23.220 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (23357 ms) 2024-03-26T02:04:23.220 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-03-26T02:04:23.220 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: attempt 0 of 20 2024-03-26T02:04:23.220 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: failed to deregister: 2024-03-26T02:04:23.220 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: { 2024-03-26T02:04:23.220 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "epoch": 19, 2024-03-26T02:04:23.220 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "modified": "2024-03-26T02:04:19.470843+0000", 2024-03-26T02:04:23.221 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "services": { 2024-03-26T02:04:23.221 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "laundry": { 2024-03-26T02:04:23.221 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "daemons": { 2024-03-26T02:04:23.221 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "summary": "", 2024-03-26T02:04:23.221 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "close-test-pid83103": { 2024-03-26T02:04:23.221 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "start_epoch": 19, 2024-03-26T02:04:23.221 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "start_stamp": "2024-03-26T02:04:18.004080+0000", 2024-03-26T02:04:23.221 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "gid": 28586, 2024-03-26T02:04:23.221 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "addr": "172.21.15.72:0/3683016774", 2024-03-26T02:04:23.221 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "metadata": { 2024-03-26T02:04:23.221 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "arch": "x86_64", 2024-03-26T02:04:23.221 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "ceph_release": "squid", 2024-03-26T02:04:23.221 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T02:04:23.221 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "ceph_version_short": "19.0.0-1775-g5e70124b", 2024-03-26T02:04:23.221 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-03-26T02:04:23.222 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "distro": "centos", 2024-03-26T02:04:23.222 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-03-26T02:04:23.222 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "distro_version": "9", 2024-03-26T02:04:23.222 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "foo": "bar", 2024-03-26T02:04:23.222 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "hostname": "smithi072", 2024-03-26T02:04:23.222 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024", 2024-03-26T02:04:23.222 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "kernel_version": "5.14.0-432.el9.x86_64", 2024-03-26T02:04:23.222 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "mem_swap_kb": "0", 2024-03-26T02:04:23.222 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "mem_total_kb": "32488116", 2024-03-26T02:04:23.222 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "os": "Linux", 2024-03-26T02:04:23.222 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "this": "that" 2024-03-26T02:04:23.222 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: }, 2024-03-26T02:04:23.222 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "task_status": {} 2024-03-26T02:04:23.222 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: }, 2024-03-26T02:04:23.223 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "pid82998": { 2024-03-26T02:04:23.223 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "start_epoch": 18, 2024-03-26T02:04:23.223 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "start_stamp": "2024-03-26T02:04:01.964107+0000", 2024-03-26T02:04:23.223 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "gid": 25163, 2024-03-26T02:04:23.223 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "addr": "172.21.15.72:0/2994750015", 2024-03-26T02:04:23.223 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "metadata": { 2024-03-26T02:04:23.223 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "arch": "x86_64", 2024-03-26T02:04:23.223 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "ceph_release": "squid", 2024-03-26T02:04:23.223 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T02:04:23.223 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "ceph_version_short": "19.0.0-1775-g5e70124b", 2024-03-26T02:04:23.223 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-03-26T02:04:23.223 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "distro": "centos", 2024-03-26T02:04:23.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-03-26T02:04:23.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "distro_version": "9", 2024-03-26T02:04:23.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "foo": "bar", 2024-03-26T02:04:23.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "hostname": "smithi072", 2024-03-26T02:04:23.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024", 2024-03-26T02:04:23.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "kernel_version": "5.14.0-432.el9.x86_64", 2024-03-26T02:04:23.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "mem_swap_kb": "0", 2024-03-26T02:04:23.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "mem_total_kb": "32488116", 2024-03-26T02:04:23.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "os": "Linux", 2024-03-26T02:04:23.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "this": "that" 2024-03-26T02:04:23.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: }, 2024-03-26T02:04:23.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "task_status": {} 2024-03-26T02:04:23.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: } 2024-03-26T02:04:23.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: } 2024-03-26T02:04:23.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: }, 2024-03-26T02:04:23.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "rgw": { 2024-03-26T02:04:23.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "daemons": { 2024-03-26T02:04:23.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "summary": "", 2024-03-26T02:04:23.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "24335": { 2024-03-26T02:04:23.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "start_epoch": 12, 2024-03-26T02:04:23.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "start_stamp": "2024-03-26T02:01:15.356767+0000", 2024-03-26T02:04:23.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "gid": 24335, 2024-03-26T02:04:23.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "addr": "172.21.15.72:0/1842098834", 2024-03-26T02:04:23.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "metadata": { 2024-03-26T02:04:23.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "arch": "x86_64", 2024-03-26T02:04:23.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "ceph_release": "squid", 2024-03-26T02:04:23.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T02:04:23.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "ceph_version_short": "19.0.0-1775-g5e70124b", 2024-03-26T02:04:23.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "container_hostname": "smithi072", 2024-03-26T02:04:23.226 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2", 2024-03-26T02:04:23.226 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-03-26T02:04:23.226 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "distro": "centos", 2024-03-26T02:04:23.226 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "distro_description": "CentOS Stream 8", 2024-03-26T02:04:23.226 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "distro_version": "8", 2024-03-26T02:04:23.226 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "frontend_config#0": "beast port=80", 2024-03-26T02:04:23.226 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "frontend_type#0": "beast", 2024-03-26T02:04:23.226 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "hostname": "smithi072", 2024-03-26T02:04:23.226 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "id": "foo.a", 2024-03-26T02:04:23.226 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024", 2024-03-26T02:04:23.226 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "kernel_version": "5.14.0-432.el9.x86_64", 2024-03-26T02:04:23.226 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "mem_swap_kb": "0", 2024-03-26T02:04:23.226 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "mem_total_kb": "32488116", 2024-03-26T02:04:23.226 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "num_handles": "1", 2024-03-26T02:04:23.227 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "os": "Linux", 2024-03-26T02:04:23.227 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "pid": "7", 2024-03-26T02:04:23.227 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "realm_id": "", 2024-03-26T02:04:23.227 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "realm_name": "", 2024-03-26T02:04:23.227 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "zone_id": "6bac1d3b-dfbc-45d3-84f4-25fe18f22adf", 2024-03-26T02:04:23.227 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "zone_name": "default", 2024-03-26T02:04:23.227 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "zonegroup_id": "65300b17-046e-40e6-a2df-ee1e799e6d65", 2024-03-26T02:04:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:23 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:04:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:23 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:23 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi072-82387-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:23 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:23 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:23 smithi173 ceph-mon[33839]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T02:04:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:23 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:23 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:04:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:23 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:23.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:23 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:04:23.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:23 smithi173 ceph-mon[33839]: pgmap v104: 816 pgs: 96 unknown, 104 creating+peering, 616 active+clean; 145 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.5 KiB/s wr, 6 op/s 2024-03-26T02:04:23.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:23 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:23.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:23 smithi173 ceph-mon[33839]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T02:04:23.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:23 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:23.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:23 smithi173 ceph-mon[33839]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T02:04:23.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:04:23.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:23.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi072-82387-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:23.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:23.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:23.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[30177]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T02:04:23.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:23.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:04:23.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:23.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:04:23.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[30177]: pgmap v104: 816 pgs: 96 unknown, 104 creating+peering, 616 active+clean; 145 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.5 KiB/s wr, 6 op/s 2024-03-26T02:04:23.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:23.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[30177]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T02:04:23.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:23.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[30177]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T02:04:23.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:04:23.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:23.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi072-82387-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:23.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:23.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:23.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[41619]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T02:04:23.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:23.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:04:23.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:23.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:04:23.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[41619]: pgmap v104: 816 pgs: 96 unknown, 104 creating+peering, 616 active+clean; 145 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.5 KiB/s wr, 6 op/s 2024-03-26T02:04:23.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:23.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[41619]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T02:04:23.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:23.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:23 smithi072 ceph-mon[41619]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T02:04:24.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:24 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:24.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:24 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:24.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:24 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/678765324' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T02:04:24.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:24 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:24.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:24 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:24.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:24 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:24.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:24 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/678765324' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T02:04:24.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:24 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:24.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:24 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:24.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:24 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:24.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:24 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/678765324' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T02:04:24.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:24 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:25.480 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:04:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:04:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:04:25.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:25 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T02:04:25.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:25 smithi072 ceph-mon[41619]: pgmap v107: 656 pgs: 160 unknown, 8 creating+peering, 488 active+clean; 145 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-26T02:04:25.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:25.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:25 smithi072 ceph-mon[41619]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T02:04:25.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:25 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:04:25.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-7"}]: dispatch 2024-03-26T02:04:25.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:25 smithi072 ceph-mon[41619]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T02:04:25.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:25 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-7"}]: dispatch 2024-03-26T02:04:25.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4288634094' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi072-82387-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:25.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1713443277' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:25.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:25 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi072-82387-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:25.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:25 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T02:04:25.973 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:25 smithi072 ceph-mon[30177]: pgmap v107: 656 pgs: 160 unknown, 8 creating+peering, 488 active+clean; 145 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-26T02:04:25.973 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:25.973 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:25 smithi072 ceph-mon[30177]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T02:04:25.973 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:25 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:04:25.973 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-7"}]: dispatch 2024-03-26T02:04:25.973 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:25 smithi072 ceph-mon[30177]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T02:04:25.973 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:25 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-7"}]: dispatch 2024-03-26T02:04:25.973 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4288634094' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi072-82387-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:25.973 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1713443277' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:25.973 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:25 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi072-82387-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:26.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:25 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T02:04:26.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:25 smithi173 ceph-mon[33839]: pgmap v107: 656 pgs: 160 unknown, 8 creating+peering, 488 active+clean; 145 MiB data, 808 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-26T02:04:26.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:26.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:25 smithi173 ceph-mon[33839]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T02:04:26.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:25 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:04:26.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-7"}]: dispatch 2024-03-26T02:04:26.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:25 smithi173 ceph-mon[33839]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T02:04:26.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:25 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-7"}]: dispatch 2024-03-26T02:04:26.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4288634094' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi072-82387-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:26.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1713443277' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:26.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:25 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi072-82387-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:26.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:26.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[41619]: Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:26.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[41619]: pgmap v109: 688 pgs: 96 creating+peering, 592 active+clean; 145 MiB data, 761 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T02:04:26.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-7"}]': finished 2024-03-26T02:04:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1713443277' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi072-82387-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[41619]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T02:04:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4261530815' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi072-83605-3"}]: dispatch 2024-03-26T02:04:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi072-83605-3"}]: dispatch 2024-03-26T02:04:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/845951764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi072-82404-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1374617939' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:04:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:04:26.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:26.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[30177]: Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:26.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[30177]: pgmap v109: 688 pgs: 96 creating+peering, 592 active+clean; 145 MiB data, 761 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T02:04:26.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-7"}]': finished 2024-03-26T02:04:26.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1713443277' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:26.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi072-82387-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:26.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[30177]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T02:04:26.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4261530815' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi072-83605-3"}]: dispatch 2024-03-26T02:04:26.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi072-83605-3"}]: dispatch 2024-03-26T02:04:26.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/845951764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi072-82404-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:26.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1374617939' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:26.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:26.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:04:26.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:26.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:26 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:04:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:26 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:26 smithi173 ceph-mon[33839]: Health check update: 10 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:26 smithi173 ceph-mon[33839]: pgmap v109: 688 pgs: 96 creating+peering, 592 active+clean; 145 MiB data, 761 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T02:04:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:26 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-7"}]': finished 2024-03-26T02:04:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:26 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1713443277' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-83037-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:26 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi072-82387-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:26 smithi173 ceph-mon[33839]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T02:04:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:26 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4261530815' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi072-83605-3"}]: dispatch 2024-03-26T02:04:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:26 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi072-83605-3"}]: dispatch 2024-03-26T02:04:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:26 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/845951764' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi072-82404-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:26 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1374617939' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:26 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:27.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:26 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:04:27.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:26 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:27.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:26 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:04:27.909 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "zonegroup_name": "defa ec_list: Running main() from gmock_main.cc 2024-03-26T02:04:27.909 INFO:tasks.workunit.client.0.smithi072.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-03-26T02:04:27.909 INFO:tasks.workunit.client.0.smithi072.stdout: ec_list: [----------] Global test environment set-up. 2024-03-26T02:04:27.910 INFO:tasks.workunit.client.0.smithi072.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-03-26T02:04:27.910 INFO:tasks.workunit.client.0.smithi072.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-03-26T02:04:27.910 INFO:tasks.workunit.client.0.smithi072.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (18043 ms) 2024-03-26T02:04:27.910 INFO:tasks.workunit.client.0.smithi072.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-03-26T02:04:27.910 INFO:tasks.workunit.client.0.smithi072.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (7554 ms) 2024-03-26T02:04:27.910 INFO:tasks.workunit.client.0.smithi072.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-03-26T02:04:27.910 INFO:tasks.workunit.client.0.smithi072.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (12847 ms) 2024-03-26T02:04:27.910 INFO:tasks.workunit.client.0.smithi072.stdout: ec_list: [----------] 3 tests from NeoradosECList (38444 ms total) 2024-03-26T02:04:27.910 INFO:tasks.workunit.client.0.smithi072.stdout: ec_list: 2024-03-26T02:04:27.910 INFO:tasks.workunit.client.0.smithi072.stdout: ec_list: [----------] Global test environment tear-down 2024-03-26T02:04:27.910 INFO:tasks.workunit.client.0.smithi072.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (38444 ms total) 2024-03-26T02:04:27.910 INFO:tasks.workunit.client.0.smithi072.stdout: ec_list: [ PASSED ] 3 tests. 2024-03-26T02:04:28.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:27 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi072-83605-3"}]': finished 2024-03-26T02:04:28.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:27 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/845951764' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi072-82404-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:28.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:27 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:28.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:27 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:04:28.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:27 smithi072 ceph-mon[41619]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T02:04:28.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:27 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4261530815' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi072-83605-3"}]: dispatch 2024-03-26T02:04:28.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:27 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-7"}]: dispatch 2024-03-26T02:04:28.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:27 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi072-83605-3"}]: dispatch 2024-03-26T02:04:28.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:27 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-7"}]: dispatch 2024-03-26T02:04:28.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:27 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:28.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:27 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi072-83605-3"}]': finished 2024-03-26T02:04:28.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:27 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/845951764' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi072-82404-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:28.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:27 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:28.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:27 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:04:28.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:27 smithi072 ceph-mon[30177]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T02:04:28.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:27 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4261530815' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi072-83605-3"}]: dispatch 2024-03-26T02:04:28.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:27 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-7"}]: dispatch 2024-03-26T02:04:28.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:27 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi072-83605-3"}]: dispatch 2024-03-26T02:04:28.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:27 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-7"}]: dispatch 2024-03-26T02:04:28.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:27 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:28.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:27 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi072-83605-3"}]': finished 2024-03-26T02:04:28.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:27 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/845951764' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi072-82404-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:28.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:27 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:28.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:27 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:04:28.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:27 smithi173 ceph-mon[33839]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T02:04:28.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:27 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4261530815' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi072-83605-3"}]: dispatch 2024-03-26T02:04:28.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:27 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-7"}]: dispatch 2024-03-26T02:04:28.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:27 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi072-83605-3"}]: dispatch 2024-03-26T02:04:28.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:27 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-7"}]: dispatch 2024-03-26T02:04:28.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:27 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:28.376 INFO:tasks.workunit.client.0.smithi072.stdout:ult" 2024-03-26T02:04:28.376 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: }, 2024-03-26T02:04:28.376 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: "task_status": {} 2024-03-26T02:04:28.376 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: } 2024-03-26T02:04:28.376 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: } 2024-03-26T02:04:28.377 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: } 2024-03-26T02:04:28.377 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: } 2024-03-26T02:04:28.377 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: } 2024-03-26T02:04:28.377 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: 2024-03-26T02:04:28.377 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: attempt 1 of 20 2024-03-26T02:04:28.377 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (10553 ms) 2024-03-26T02:04:28.377 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (39075 ms total) 2024-03-26T02:04:28.377 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: 2024-03-26T02:04:28.377 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: [----------] Global test environment tear-down 2024-03-26T02:04:28.377 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (39075 ms total) 2024-03-26T02:04:28.377 INFO:tasks.workunit.client.0.smithi072.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-03-26T02:04:29.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:28 smithi072 ceph-mon[41619]: pgmap v112: 712 pgs: 160 unknown, 32 creating+peering, 520 active+clean; 145 MiB data, 761 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 4 op/s 2024-03-26T02:04:29.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:28 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi072-83605-3"}]': finished 2024-03-26T02:04:29.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:28 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-7"}]': finished 2024-03-26T02:04:29.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:28 smithi072 ceph-mon[41619]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T02:04:29.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:28 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2463912934' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi072-82387-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:29.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:28 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi072-82387-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:29.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:28 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:29.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:28 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4044669520' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T02:04:29.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:28 smithi072 ceph-mon[30177]: pgmap v112: 712 pgs: 160 unknown, 32 creating+peering, 520 active+clean; 145 MiB data, 761 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 4 op/s 2024-03-26T02:04:29.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:28 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi072-83605-3"}]': finished 2024-03-26T02:04:29.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:28 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-7"}]': finished 2024-03-26T02:04:29.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:28 smithi072 ceph-mon[30177]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T02:04:29.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:28 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2463912934' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi072-82387-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:29.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:28 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi072-82387-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:29.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:28 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:29.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:28 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4044669520' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T02:04:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:28 smithi173 ceph-mon[33839]: pgmap v112: 712 pgs: 160 unknown, 32 creating+peering, 520 active+clean; 145 MiB data, 761 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 4 op/s 2024-03-26T02:04:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:28 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi072-83605-3"}]': finished 2024-03-26T02:04:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:28 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-7"}]': finished 2024-03-26T02:04:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:28 smithi173 ceph-mon[33839]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T02:04:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:28 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2463912934' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi072-82387-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:28 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi072-82387-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:28 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:28 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4044669520' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T02:04:30.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:30 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T02:04:30.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi072-82387-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:30.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:30 smithi072 ceph-mon[41619]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T02:04:30.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3712270916' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi072-82404-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:30.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:30.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4"}]: dispatch 2024-03-26T02:04:30.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:30.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:30 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T02:04:30.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi072-82387-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:30.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:30 smithi072 ceph-mon[30177]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T02:04:30.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3712270916' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi072-82404-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:30.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:30.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4"}]: dispatch 2024-03-26T02:04:30.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:30.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:30 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T02:04:30.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi072-82387-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:30.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:30 smithi173 ceph-mon[33839]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T02:04:30.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3712270916' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi072-82404-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:30.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:30.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4"}]: dispatch 2024-03-26T02:04:30.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:31.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:04:30 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:04:31.585 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-03-26T02:04:31.585 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-03-26T02:04:31.585 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-03-26T02:04:31.585 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-03-26T02:04:31.585 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-03-26T02:04:31.586 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-03-26T02:04:31.586 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-03-26T02:04:31.586 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: 2024-03-26T02:04:31.586 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-03-26T02:04:31.586 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-03-26T02:04:31.586 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (11365 ms) 2024-03-26T02:04:31.586 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-03-26T02:04:31.586 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (5128 ms) 2024-03-26T02:04:31.586 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-03-26T02:04:31.586 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (4055 ms) 2024-03-26T02:04:31.586 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-03-26T02:04:31.586 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (2789 ms) 2024-03-26T02:04:31.586 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-03-26T02:04:31.587 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (4938 ms) 2024-03-26T02:04:31.587 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-03-26T02:04:31.587 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (4764 ms) 2024-03-26T02:04:31.587 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-03-26T02:04:31.587 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (9286 ms) 2024-03-26T02:04:31.587 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (42325 ms total) 2024-03-26T02:04:31.587 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: 2024-03-26T02:04:31.587 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-03-26T02:04:31.587 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (42325 ms total) 2024-03-26T02:04:31.587 INFO:tasks.workunit.client.0.smithi072.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-03-26T02:04:31.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:31 smithi072 ceph-mon[41619]: pgmap v115: 680 pgs: 160 unknown, 32 creating+peering, 488 active+clean; 145 MiB data, 761 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:04:31.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:31 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3712270916' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi072-82404-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:31.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:31 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4"}]': finished 2024-03-26T02:04:31.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:31 smithi072 ceph-mon[41619]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T02:04:31.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:31 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4", "tierpool": "test-rados-api-smithi072-82938-6"}]: dispatch 2024-03-26T02:04:31.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:31 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:31.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:31 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:31.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:31 smithi072 ceph-mon[41619]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:31.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:31 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:04:31.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:31 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:04:31.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:31 smithi072 ceph-mon[30177]: pgmap v115: 680 pgs: 160 unknown, 32 creating+peering, 488 active+clean; 145 MiB data, 761 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:04:31.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:31 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3712270916' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi072-82404-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:31.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:31 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4"}]': finished 2024-03-26T02:04:31.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:31 smithi072 ceph-mon[30177]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T02:04:31.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:31 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4", "tierpool": "test-rados-api-smithi072-82938-6"}]: dispatch 2024-03-26T02:04:31.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:31 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:31.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:31 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:31.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:31 smithi072 ceph-mon[30177]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:31.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:31 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:04:31.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:31 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:04:31.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:31 smithi173 ceph-mon[33839]: pgmap v115: 680 pgs: 160 unknown, 32 creating+peering, 488 active+clean; 145 MiB data, 761 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:04:31.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:31 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3712270916' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi072-82404-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:31.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:31 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4"}]': finished 2024-03-26T02:04:31.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:31 smithi173 ceph-mon[33839]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T02:04:31.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:31 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4", "tierpool": "test-rados-api-smithi072-82938-6"}]: dispatch 2024-03-26T02:04:31.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:31 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:31.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:31 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:31.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:31 smithi173 ceph-mon[33839]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:31.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:31 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:04:31.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:31 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:04:32.838 INFO:tasks.workunit.client.0.smithi072.stdout:watch_notify_pp: flushed 2024-03-26T02:04:32.838 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3006 ms) 2024-03-26T02:04:32.838 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-03-26T02:04:32.838 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: trying... 2024-03-26T02:04:32.838 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: handle_notify cookie 94113618205600 notify_id 403726925828 notifier_gid 15249 2024-03-26T02:04:32.838 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: timed out 2024-03-26T02:04:32.838 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: flushing 2024-03-26T02:04:32.838 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: flushed 2024-03-26T02:04:32.838 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3005 ms) 2024-03-26T02:04:32.838 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-03-26T02:04:32.838 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: List watches 2024-03-26T02:04:32.838 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: notify2 2024-03-26T02:04:32.838 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: handle_notify cookie 94113618205600 notify_id 416611827717 notifier_gid 15249 2024-03-26T02:04:32.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: notify2 done 2024-03-26T02:04:32.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: watch_check 2024-03-26T02:04:32.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: unwatch2 2024-03-26T02:04:32.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: flushing 2024-03-26T02:04:32.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: done 2024-03-26T02:04:32.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3027 ms) 2024-03-26T02:04:32.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-03-26T02:04:32.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: List watches 2024-03-26T02:04:32.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: notify2 2024-03-26T02:04:32.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: handle_notify cookie 94113618205600 notify_id 429496729605 notifier_gid 15249 2024-03-26T02:04:32.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: notify2 done 2024-03-26T02:04:32.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: watch_check 2024-03-26T02:04:32.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: unwatch2 2024-03-26T02:04:32.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: flushing 2024-03-26T02:04:32.840 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: done 2024-03-26T02:04:32.840 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3093 ms) 2024-03-26T02:04:32.840 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (17187 ms total) 2024-03-26T02:04:32.840 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: 2024-03-26T02:04:32.840 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-03-26T02:04:32.840 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (43600 ms total) 2024-03-26T02:04:32.840 INFO:tasks.workunit.client.0.smithi072.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-03-26T02:04:33.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4", "tierpool": "test-rados-api-smithi072-82938-6"}]': finished 2024-03-26T02:04:33.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:32 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:33.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:32 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:04:33.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:33.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:33.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:32 smithi173 ceph-mon[33839]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T02:04:33.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi072-82938-6", "pool2": "test-rados-api-smithi072-82938-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T02:04:33.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:32 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:33.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi072-82938-6", "pool2": "test-rados-api-smithi072-82938-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T02:04:33.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:32 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:04:33.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:32 smithi173 ceph-mon[33839]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T02:04:33.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-9"}]: dispatch 2024-03-26T02:04:33.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:32 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-9"}]: dispatch 2024-03-26T02:04:33.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3365157457' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:33.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:32 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:33.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2462023807' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi072-82387-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:33.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:32 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi072-82387-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:33.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:33.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4", "tierpool": "test-rados-api-smithi072-82938-6"}]': finished 2024-03-26T02:04:33.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:33.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:04:33.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:33.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:33.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[30177]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T02:04:33.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi072-82938-6", "pool2": "test-rados-api-smithi072-82938-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T02:04:33.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:33.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi072-82938-6", "pool2": "test-rados-api-smithi072-82938-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T02:04:33.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:04:33.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[30177]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T02:04:33.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-9"}]: dispatch 2024-03-26T02:04:33.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-9"}]: dispatch 2024-03-26T02:04:33.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3365157457' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:33.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:33.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2462023807' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi072-82387-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:33.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi072-82387-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:33.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:33.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi072-82938-4", "tierpool": "test-rados-api-smithi072-82938-6"}]': finished 2024-03-26T02:04:33.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:33.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:04:33.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:33.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:33.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[41619]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T02:04:33.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi072-82938-6", "pool2": "test-rados-api-smithi072-82938-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T02:04:33.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:33.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3349307567' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi072-82938-6", "pool2": "test-rados-api-smithi072-82938-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T02:04:33.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:04:33.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[41619]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T02:04:33.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-9"}]: dispatch 2024-03-26T02:04:33.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-9"}]: dispatch 2024-03-26T02:04:33.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3365157457' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:33.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:33.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2462023807' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi072-82387-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:33.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi072-82387-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:33.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:33.804 INFO:tasks.workunit.client.0.smithi072.stdout: write_operations: Running main() from gmock_main.cc 2024-03-26T02:04:33.804 INFO:tasks.workunit.client.0.smithi072.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-03-26T02:04:33.804 INFO:tasks.workunit.client.0.smithi072.stdout: write_operations: [----------] Global test environment set-up. 2024-03-26T02:04:33.804 INFO:tasks.workunit.client.0.smithi072.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-03-26T02:04:33.804 INFO:tasks.workunit.client.0.smithi072.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-03-26T02:04:33.804 INFO:tasks.workunit.client.0.smithi072.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (10857 ms) 2024-03-26T02:04:33.804 INFO:tasks.workunit.client.0.smithi072.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-03-26T02:04:33.804 INFO:tasks.workunit.client.0.smithi072.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (8443 ms) 2024-03-26T02:04:33.804 INFO:tasks.workunit.client.0.smithi072.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-03-26T02:04:33.804 INFO:tasks.workunit.client.0.smithi072.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (3052 ms) 2024-03-26T02:04:33.805 INFO:tasks.workunit.client.0.smithi072.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-03-26T02:04:33.805 INFO:tasks.workunit.client.0.smithi072.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (4327 ms) 2024-03-26T02:04:33.805 INFO:tasks.workunit.client.0.smithi072.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-03-26T02:04:33.805 INFO:tasks.workunit.client.0.smithi072.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (3975 ms) 2024-03-26T02:04:33.805 INFO:tasks.workunit.client.0.smithi072.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-03-26T02:04:33.805 INFO:tasks.workunit.client.0.smithi072.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (4475 ms) 2024-03-26T02:04:33.805 INFO:tasks.workunit.client.0.smithi072.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-03-26T02:04:33.805 INFO:tasks.workunit.client.0.smithi072.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (9176 ms) 2024-03-26T02:04:33.805 INFO:tasks.workunit.client.0.smithi072.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (44305 ms total) 2024-03-26T02:04:33.805 INFO:tasks.workunit.client.0.smithi072.stdout: write_operations: 2024-03-26T02:04:33.805 INFO:tasks.workunit.client.0.smithi072.stdout: write_operations: [----------] Global test environment tear-down 2024-03-26T02:04:33.805 INFO:tasks.workunit.client.0.smithi072.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (44305 ms total) 2024-03-26T02:04:33.805 INFO:tasks.workunit.client.0.smithi072.stdout: write_operations: [ PASSED ] 7 tests. 2024-03-26T02:04:33.837 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-03-26T02:04:33.837 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [----------] Global test environment set-up. 2024-03-26T02:04:33.837 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-03-26T02:04:33.837 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-03-26T02:04:33.837 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-03-26T02:04:33.837 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (1 ms total) 2024-03-26T02:04:33.837 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: 2024-03-26T02:04:33.838 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-03-26T02:04:33.838 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-03-26T02:04:33.838 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-03-26T02:04:33.838 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: 2024-03-26T02:03:49.168+0000 7fd5f9292780 -1 failed for service _ceph-mon._tcp 2024-03-26T02:04:33.838 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: 2024-03-26T02:03:49.168+0000 7fd5f9292780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-03-26T02:04:33.838 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (51 ms) 2024-03-26T02:04:33.838 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-03-26T02:04:33.838 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2012 ms) 2024-03-26T02:04:33.838 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2063 ms total) 2024-03-26T02:04:33.838 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: 2024-03-26T02:04:33.838 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-03-26T02:04:33.838 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-03-26T02:04:33.838 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: started 0x55dc4e49f600 2024-03-26T02:04:33.838 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: started 0x55dc4e6a1020 2024-03-26T02:04:33.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: started 2 aios 2024-03-26T02:04:33.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: waiting 0x55dc4e49f600 2024-03-26T02:04:33.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: waiting 0x55dc4e6a1020 2024-03-26T02:04:33.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: done. 2024-03-26T02:04:33.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (12289 ms) 2024-03-26T02:04:33.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (12289 ms total) 2024-03-26T02:04:33.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: 2024-03-26T02:04:33.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-03-26T02:04:33.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-03-26T02:04:33.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-03-26T02:04:33.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-03-26T02:04:33.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [ OK ] LibRadosMisc.Exec (65 ms) 2024-03-26T02:04:33.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-03-26T02:04:33.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (14 ms) 2024-03-26T02:04:33.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-03-26T02:04:33.839 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (8 ms) 2024-03-26T02:04:33.840 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-03-26T02:04:33.840 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [ OK ] LibRadosMisc.Applications (6005 ms) 2024-03-26T02:04:33.840 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-03-26T02:04:33.840 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-03-26T02:04:33.840 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-03-26T02:04:33.840 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-03-26T02:04:33.840 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-03-26T02:04:33.840 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (21014 ms) 2024-03-26T02:04:33.840 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [----------] 8 tests from LibRadosMisc (27107 ms total) 2024-03-26T02:04:33.840 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: 2024-03-26T02:04:33.840 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [----------] Global test environment tear-down 2024-03-26T02:04:33.840 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (44698 ms total) 2024-03-26T02:04:33.840 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc: [ PASSED ] 12 tests. 2024-03-26T02:04:34.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: pgmap v118: 712 pgs: 32 unknown, 96 creating+peering, 584 active+clean; 145 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 5.0 KiB/s wr, 10 op/s 2024-03-26T02:04:34.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-03-26T02:04:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-03-26T02:04:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-03-26T02:04:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-03-26T02:04:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-03-26T02:04:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-03-26T02:04:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-03-26T02:04:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-03-26T02:04:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-03-26T02:04:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-03-26T02:04:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-03-26T02:04:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-03-26T02:04:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-03-26T02:04:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-03-26T02:04:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-03-26T02:04:34.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-03-26T02:04:34.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-03-26T02:04:34.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-03-26T02:04:34.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-03-26T02:04:34.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-03-26T02:04:34.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: 15.3 deep-scrub starts 2024-03-26T02:04:34.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: 15.3 deep-scrub ok 2024-03-26T02:04:34.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: 15.6 deep-scrub starts 2024-03-26T02:04:34.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: 15.6 deep-scrub ok 2024-03-26T02:04:34.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: 15.9 deep-scrub starts 2024-03-26T02:04:34.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: 15.9 deep-scrub ok 2024-03-26T02:04:34.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: 15.7 deep-scrub starts 2024-03-26T02:04:34.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: 15.7 deep-scrub ok 2024-03-26T02:04:34.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-9"}]': finished 2024-03-26T02:04:34.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:34.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi072-82387-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:34.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: osdmap e107: 8 total, 8 up, 8 in 2024-03-26T02:04:34.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-9", "mode": "writeback"}]: dispatch 2024-03-26T02:04:34.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/15467834' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi072-82404-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:34.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:34.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-9", "mode": "writeback"}]: dispatch 2024-03-26T02:04:34.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:33 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi072-82404-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:34.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: pgmap v118: 712 pgs: 32 unknown, 96 creating+peering, 584 active+clean; 145 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 5.0 KiB/s wr, 10 op/s 2024-03-26T02:04:34.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-03-26T02:04:34.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-03-26T02:04:34.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-03-26T02:04:34.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-03-26T02:04:34.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-03-26T02:04:34.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-03-26T02:04:34.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-03-26T02:04:34.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-03-26T02:04:34.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-03-26T02:04:34.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-03-26T02:04:34.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-03-26T02:04:34.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-03-26T02:04:34.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-03-26T02:04:34.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-03-26T02:04:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-03-26T02:04:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-03-26T02:04:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-03-26T02:04:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-03-26T02:04:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-03-26T02:04:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-03-26T02:04:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: 15.3 deep-scrub starts 2024-03-26T02:04:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: 15.3 deep-scrub ok 2024-03-26T02:04:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: 15.6 deep-scrub starts 2024-03-26T02:04:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: 15.6 deep-scrub ok 2024-03-26T02:04:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: 15.9 deep-scrub starts 2024-03-26T02:04:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: 15.9 deep-scrub ok 2024-03-26T02:04:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: 15.7 deep-scrub starts 2024-03-26T02:04:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: 15.7 deep-scrub ok 2024-03-26T02:04:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-9"}]': finished 2024-03-26T02:04:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:34.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi072-82387-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:34.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: osdmap e107: 8 total, 8 up, 8 in 2024-03-26T02:04:34.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-9", "mode": "writeback"}]: dispatch 2024-03-26T02:04:34.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/15467834' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi072-82404-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:34.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:34.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-9", "mode": "writeback"}]: dispatch 2024-03-26T02:04:34.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi072-82404-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:34.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: pgmap v118: 712 pgs: 32 unknown, 96 creating+peering, 584 active+clean; 145 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 5.0 KiB/s wr, 10 op/s 2024-03-26T02:04:34.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-03-26T02:04:34.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.0"}]: dispatch 2024-03-26T02:04:34.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-03-26T02:04:34.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.1"}]: dispatch 2024-03-26T02:04:34.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-03-26T02:04:34.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.2"}]: dispatch 2024-03-26T02:04:34.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-03-26T02:04:34.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.3"}]: dispatch 2024-03-26T02:04:34.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-03-26T02:04:34.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.4"}]: dispatch 2024-03-26T02:04:34.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-03-26T02:04:34.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.5"}]: dispatch 2024-03-26T02:04:34.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-03-26T02:04:34.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.6"}]: dispatch 2024-03-26T02:04:34.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-03-26T02:04:34.223 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.7"}]: dispatch 2024-03-26T02:04:34.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-03-26T02:04:34.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.8"}]: dispatch 2024-03-26T02:04:34.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-03-26T02:04:34.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "15.9"}]: dispatch 2024-03-26T02:04:34.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: 15.3 deep-scrub starts 2024-03-26T02:04:34.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: 15.3 deep-scrub ok 2024-03-26T02:04:34.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: 15.6 deep-scrub starts 2024-03-26T02:04:34.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: 15.6 deep-scrub ok 2024-03-26T02:04:34.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: 15.9 deep-scrub starts 2024-03-26T02:04:34.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: 15.9 deep-scrub ok 2024-03-26T02:04:34.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: 15.7 deep-scrub starts 2024-03-26T02:04:34.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: 15.7 deep-scrub ok 2024-03-26T02:04:34.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-9"}]': finished 2024-03-26T02:04:34.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:34.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi072-82387-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:34.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: osdmap e107: 8 total, 8 up, 8 in 2024-03-26T02:04:34.225 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-9", "mode": "writeback"}]: dispatch 2024-03-26T02:04:34.225 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/15467834' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi072-82404-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:34.225 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:34.225 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-9", "mode": "writeback"}]: dispatch 2024-03-26T02:04:34.225 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:33 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi072-82404-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:35.018 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:04:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:04:35.019 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[30177]: 15.8 deep-scrub starts 2024-03-26T02:04:35.019 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[30177]: 15.8 deep-scrub ok 2024-03-26T02:04:35.019 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[30177]: pgmap v121: 708 pgs: 192 unknown, 32 creating+peering, 484 active+clean; 145 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T02:04:35.019 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[30177]: 15.5 deep-scrub starts 2024-03-26T02:04:35.019 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[30177]: 15.5 deep-scrub ok 2024-03-26T02:04:35.019 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[30177]: 15.4 deep-scrub starts 2024-03-26T02:04:35.019 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[30177]: 15.4 deep-scrub ok 2024-03-26T02:04:35.019 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:04:35.020 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-9", "mode": "writeback"}]': finished 2024-03-26T02:04:35.020 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi072-82404-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:35.020 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[30177]: osdmap e108: 8 total, 8 up, 8 in 2024-03-26T02:04:35.020 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:04:35.020 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:35.020 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi072-82728-10"}]: dispatch 2024-03-26T02:04:35.020 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi072-82728-10"}]: dispatch 2024-03-26T02:04:35.020 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi072-82728-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:35.020 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:04:35.020 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[41619]: 15.8 deep-scrub starts 2024-03-26T02:04:35.020 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[41619]: 15.8 deep-scrub ok 2024-03-26T02:04:35.020 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[41619]: pgmap v121: 708 pgs: 192 unknown, 32 creating+peering, 484 active+clean; 145 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T02:04:35.021 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[41619]: 15.5 deep-scrub starts 2024-03-26T02:04:35.021 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[41619]: 15.5 deep-scrub ok 2024-03-26T02:04:35.021 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[41619]: 15.4 deep-scrub starts 2024-03-26T02:04:35.021 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[41619]: 15.4 deep-scrub ok 2024-03-26T02:04:35.021 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:04:35.021 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-9", "mode": "writeback"}]': finished 2024-03-26T02:04:35.021 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi072-82404-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:35.021 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[41619]: osdmap e108: 8 total, 8 up, 8 in 2024-03-26T02:04:35.021 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:04:35.021 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:35.021 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi072-82728-10"}]: dispatch 2024-03-26T02:04:35.021 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi072-82728-10"}]: dispatch 2024-03-26T02:04:35.021 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi072-82728-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:35.021 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:34 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:04:35.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:34 smithi173 ceph-mon[33839]: 15.8 deep-scrub starts 2024-03-26T02:04:35.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:34 smithi173 ceph-mon[33839]: 15.8 deep-scrub ok 2024-03-26T02:04:35.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:34 smithi173 ceph-mon[33839]: pgmap v121: 708 pgs: 192 unknown, 32 creating+peering, 484 active+clean; 145 MiB data, 729 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T02:04:35.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:34 smithi173 ceph-mon[33839]: 15.5 deep-scrub starts 2024-03-26T02:04:35.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:34 smithi173 ceph-mon[33839]: 15.5 deep-scrub ok 2024-03-26T02:04:35.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:34 smithi173 ceph-mon[33839]: 15.4 deep-scrub starts 2024-03-26T02:04:35.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:34 smithi173 ceph-mon[33839]: 15.4 deep-scrub ok 2024-03-26T02:04:35.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:34 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:04:35.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:34 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-9", "mode": "writeback"}]': finished 2024-03-26T02:04:35.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:34 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi072-82404-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:35.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:34 smithi173 ceph-mon[33839]: osdmap e108: 8 total, 8 up, 8 in 2024-03-26T02:04:35.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:34 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:04:35.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:34 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:35.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:34 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi072-82728-10"}]: dispatch 2024-03-26T02:04:35.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:34 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi072-82728-10"}]: dispatch 2024-03-26T02:04:35.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:34 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi072-82728-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:35.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:34 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:04:36.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:35 smithi173 ceph-mon[33839]: 15.0 deep-scrub starts 2024-03-26T02:04:36.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:35 smithi173 ceph-mon[33839]: 15.0 deep-scrub ok 2024-03-26T02:04:36.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:35 smithi173 ceph-mon[33839]: 15.1 deep-scrub starts 2024-03-26T02:04:36.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:35 smithi173 ceph-mon[33839]: 15.1 deep-scrub ok 2024-03-26T02:04:36.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:35 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi072-82728-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:36.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:35 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:04:36.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:35 smithi173 ceph-mon[33839]: osdmap e109: 8 total, 8 up, 8 in 2024-03-26T02:04:36.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:35 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-9"}]: dispatch 2024-03-26T02:04:36.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:35 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi072-82728-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi072-82728-10"}]: dispatch 2024-03-26T02:04:36.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:35 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2330030367' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi072-82387-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:36.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:35 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-9"}]: dispatch 2024-03-26T02:04:36.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:35 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi072-82387-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:36.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:35 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:36.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:35 smithi173 ceph-mon[33839]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:36.129 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[30177]: 15.0 deep-scrub starts 2024-03-26T02:04:36.129 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[30177]: 15.0 deep-scrub ok 2024-03-26T02:04:36.129 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[30177]: 15.1 deep-scrub starts 2024-03-26T02:04:36.129 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[30177]: 15.1 deep-scrub ok 2024-03-26T02:04:36.130 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi072-82728-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:36.130 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:04:36.130 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[30177]: osdmap e109: 8 total, 8 up, 8 in 2024-03-26T02:04:36.130 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-9"}]: dispatch 2024-03-26T02:04:36.130 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi072-82728-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi072-82728-10"}]: dispatch 2024-03-26T02:04:36.130 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2330030367' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi072-82387-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:36.130 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-9"}]: dispatch 2024-03-26T02:04:36.130 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi072-82387-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:36.130 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:36.130 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[30177]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:36.131 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[41619]: 15.0 deep-scrub starts 2024-03-26T02:04:36.131 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[41619]: 15.0 deep-scrub ok 2024-03-26T02:04:36.131 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[41619]: 15.1 deep-scrub starts 2024-03-26T02:04:36.132 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[41619]: 15.1 deep-scrub ok 2024-03-26T02:04:36.132 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi072-82728-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:36.132 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:04:36.132 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[41619]: osdmap e109: 8 total, 8 up, 8 in 2024-03-26T02:04:36.132 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-9"}]: dispatch 2024-03-26T02:04:36.132 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi072-82728-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi072-82728-10"}]: dispatch 2024-03-26T02:04:36.132 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2330030367' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi072-82387-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:36.132 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-9"}]: dispatch 2024-03-26T02:04:36.132 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi072-82387-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:36.132 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:36.132 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:35 smithi072 ceph-mon[41619]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:37.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:36 smithi173 ceph-mon[33839]: 15.2 deep-scrub starts 2024-03-26T02:04:37.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:36 smithi173 ceph-mon[33839]: 15.2 deep-scrub ok 2024-03-26T02:04:37.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:36 smithi173 ceph-mon[33839]: pgmap v124: 516 pgs: 64 creating+peering, 452 active+clean; 217 MiB data, 998 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:04:37.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:36 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:04:37.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:36 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-9"}]': finished 2024-03-26T02:04:37.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:36 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi072-82387-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:37.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:36 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:37.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:36 smithi173 ceph-mon[33839]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T02:04:37.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:36 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/150563177' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi072-82404-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:37.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:36 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi072-82404-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:37.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:36 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:04:37.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:36 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:04:37.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:36 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:04:37.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:36 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:04:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[30177]: 15.2 deep-scrub starts 2024-03-26T02:04:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[30177]: 15.2 deep-scrub ok 2024-03-26T02:04:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[30177]: pgmap v124: 516 pgs: 64 creating+peering, 452 active+clean; 217 MiB data, 998 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:04:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:04:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-9"}]': finished 2024-03-26T02:04:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi072-82387-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:37.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:37.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[30177]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T02:04:37.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/150563177' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi072-82404-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:37.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi072-82404-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:37.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:04:37.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:04:37.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:04:37.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:04:37.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[41619]: 15.2 deep-scrub starts 2024-03-26T02:04:37.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[41619]: 15.2 deep-scrub ok 2024-03-26T02:04:37.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[41619]: pgmap v124: 516 pgs: 64 creating+peering, 452 active+clean; 217 MiB data, 998 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:04:37.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:04:37.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-9"}]': finished 2024-03-26T02:04:37.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi072-82387-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:37.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:37.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[41619]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T02:04:37.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/150563177' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi072-82404-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:37.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi072-82404-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:37.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:04:37.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:04:37.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:04:37.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:36 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:04:38.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:37 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi072-82728-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi072-82728-10"}]': finished 2024-03-26T02:04:38.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:37 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi072-82404-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:38.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:37 smithi072 ceph-mon[30177]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T02:04:38.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:37 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2463762264' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:38.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:37 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:38.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:37 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:38.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:37 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi072-82728-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi072-82728-10"}]': finished 2024-03-26T02:04:38.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:37 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi072-82404-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:38.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:37 smithi072 ceph-mon[41619]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T02:04:38.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:37 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2463762264' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:38.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:37 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:38.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:37 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:38.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:37 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi072-82728-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi072-82728-10"}]': finished 2024-03-26T02:04:38.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:37 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi072-82404-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:38.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:37 smithi173 ceph-mon[33839]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T02:04:38.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:37 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2463762264' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:38.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:37 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:38.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:37 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:39.120 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[30177]: pgmap v127: 492 pgs: 104 unknown, 388 active+clean; 217 MiB data, 998 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:04:39.120 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:04:39.120 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:39.120 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[30177]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T02:04:39.120 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2337471269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi072-82387-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:39.120 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:39.120 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:39.120 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:04:39.120 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi072-82387-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:39.121 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:39.121 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:04:39.121 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:04:39.121 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:04:39.121 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi072-82387-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:39.121 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:39.121 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[30177]: osdmap e113: 8 total, 8 up, 8 in 2024-03-26T02:04:39.121 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[41619]: pgmap v127: 492 pgs: 104 unknown, 388 active+clean; 217 MiB data, 998 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:04:39.121 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:04:39.121 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:39.121 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[41619]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T02:04:39.121 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2337471269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi072-82387-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:39.122 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:39.122 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:39.122 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:04:39.122 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi072-82387-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:39.122 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:39.122 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:04:39.122 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:04:39.122 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:04:39.122 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi072-82387-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:39.122 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:39.122 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:38 smithi072 ceph-mon[41619]: osdmap e113: 8 total, 8 up, 8 in 2024-03-26T02:04:39.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:38 smithi173 ceph-mon[33839]: pgmap v127: 492 pgs: 104 unknown, 388 active+clean; 217 MiB data, 998 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:04:39.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:38 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:04:39.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:38 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82643-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:39.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:38 smithi173 ceph-mon[33839]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T02:04:39.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:38 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2337471269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi072-82387-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:39.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:38 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:39.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:38 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:39.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:38 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:04:39.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:38 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi072-82387-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:39.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:38 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:39.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:38 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:04:39.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:38 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:04:39.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:38 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:04:39.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:38 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi072-82387-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:39.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:38 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:39.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:38 smithi173 ceph-mon[33839]: osdmap e113: 8 total, 8 up, 8 in 2024-03-26T02:04:39.839 INFO:tasks.workunit.client.0.smithi072.stdout: misc: Running main() from gmock_main.cc 2024-03-26T02:04:39.839 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-03-26T02:04:39.839 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [----------] Global test environment set-up. 2024-03-26T02:04:39.839 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-03-26T02:04:39.839 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-03-26T02:04:39.839 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [ OK ] NeoRadosMisc.Version (9989 ms) 2024-03-26T02:04:39.839 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-03-26T02:04:39.839 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (1728 ms) 2024-03-26T02:04:39.839 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-03-26T02:04:39.839 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [ OK ] NeoRadosMisc.LongName (4580 ms) 2024-03-26T02:04:39.839 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-03-26T02:04:39.839 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (5832 ms) 2024-03-26T02:04:39.840 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-03-26T02:04:39.840 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (4601 ms) 2024-03-26T02:04:39.840 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-03-26T02:04:39.840 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (6077 ms) 2024-03-26T02:04:39.840 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-03-26T02:04:39.840 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [ OK ] NeoRadosMisc.Exec (3486 ms) 2024-03-26T02:04:39.840 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-03-26T02:04:39.840 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (3171 ms) 2024-03-26T02:04:39.840 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-03-26T02:04:39.840 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (2685 ms) 2024-03-26T02:04:39.840 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-03-26T02:04:39.840 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [ OK ] NeoRadosMisc.BigObject (3213 ms) 2024-03-26T02:04:39.840 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-03-26T02:04:39.840 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2285 ms) 2024-03-26T02:04:39.840 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-03-26T02:04:39.841 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (2739 ms) 2024-03-26T02:04:39.841 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [----------] 12 tests from NeoRadosMisc (50386 ms total) 2024-03-26T02:04:39.841 INFO:tasks.workunit.client.0.smithi072.stdout: misc: 2024-03-26T02:04:39.841 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [----------] Global test environment tear-down 2024-03-26T02:04:39.841 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [==========] 12 tests from 1 test suite ran. (50386 ms total) 2024-03-26T02:04:39.841 INFO:tasks.workunit.client.0.smithi072.stdout: misc: [ PASSED ] 12 tests. 2024-03-26T02:04:40.134 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2463762264' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82643-10", "tierpool":"test-rados-api-smithi072-82643-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T02:04:40.134 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:39 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82643-10", "tierpool":"test-rados-api-smithi072-82643-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T02:04:40.134 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:40.134 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:39 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82643-10", "tierpool":"test-rados-api-smithi072-82643-10-cache", "force_nonempty":""}]': finished 2024-03-26T02:04:40.134 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:39 smithi072 ceph-mon[30177]: osdmap e114: 8 total, 8 up, 8 in 2024-03-26T02:04:40.134 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2463762264' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82643-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:40.134 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1357416515' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi072-82404-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:40.134 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:39 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82643-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:40.135 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:39 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi072-82404-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:40.135 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2463762264' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82643-10", "tierpool":"test-rados-api-smithi072-82643-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T02:04:40.135 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:39 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82643-10", "tierpool":"test-rados-api-smithi072-82643-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T02:04:40.135 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:40.135 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:39 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82643-10", "tierpool":"test-rados-api-smithi072-82643-10-cache", "force_nonempty":""}]': finished 2024-03-26T02:04:40.135 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:39 smithi072 ceph-mon[41619]: osdmap e114: 8 total, 8 up, 8 in 2024-03-26T02:04:40.135 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2463762264' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82643-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:40.135 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1357416515' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi072-82404-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:40.135 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:39 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82643-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:40.135 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:39 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi072-82404-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:40.214 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2463762264' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82643-10", "tierpool":"test-rados-api-smithi072-82643-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T02:04:40.214 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:39 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82643-10", "tierpool":"test-rados-api-smithi072-82643-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T02:04:40.214 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:40.214 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:39 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82643-10", "tierpool":"test-rados-api-smithi072-82643-10-cache", "force_nonempty":""}]': finished 2024-03-26T02:04:40.214 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:39 smithi173 ceph-mon[33839]: osdmap e114: 8 total, 8 up, 8 in 2024-03-26T02:04:40.214 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2463762264' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82643-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:40.214 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1357416515' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi072-82404-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:40.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:39 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82643-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:40.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:39 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi072-82404-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:40.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:04:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:04:41.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[30177]: pgmap v130: 588 pgs: 200 unknown, 388 active+clean; 217 MiB data, 998 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:04:41.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:41.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:41.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:41.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[30177]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:41.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82643-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:41.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi072-82404-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:41.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:04:41.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[30177]: osdmap e115: 8 total, 8 up, 8 in 2024-03-26T02:04:41.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-11"}]: dispatch 2024-03-26T02:04:41.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-11"}]: dispatch 2024-03-26T02:04:41.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3953820198' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi072-82387-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:41.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2463762264' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82643-10", "tierpool":"test-rados-api-smithi072-82643-10-cache"}]: dispatch 2024-03-26T02:04:41.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82643-10", "tierpool":"test-rados-api-smithi072-82643-10-cache"}]: dispatch 2024-03-26T02:04:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[41619]: pgmap v130: 588 pgs: 200 unknown, 388 active+clean; 217 MiB data, 998 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:04:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:41.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:41.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:41.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[41619]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:41.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82643-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:41.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi072-82404-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:41.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:04:41.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[41619]: osdmap e115: 8 total, 8 up, 8 in 2024-03-26T02:04:41.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-11"}]: dispatch 2024-03-26T02:04:41.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-11"}]: dispatch 2024-03-26T02:04:41.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3953820198' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi072-82387-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:41.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2463762264' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82643-10", "tierpool":"test-rados-api-smithi072-82643-10-cache"}]: dispatch 2024-03-26T02:04:41.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:40 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82643-10", "tierpool":"test-rados-api-smithi072-82643-10-cache"}]: dispatch 2024-03-26T02:04:41.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:40 smithi173 ceph-mon[33839]: pgmap v130: 588 pgs: 200 unknown, 388 active+clean; 217 MiB data, 998 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:04:41.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:41.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:41.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:40 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:41.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:40 smithi173 ceph-mon[33839]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:41.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:40 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82643-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:41.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:40 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi072-82404-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:41.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:40 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:04:41.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:40 smithi173 ceph-mon[33839]: osdmap e115: 8 total, 8 up, 8 in 2024-03-26T02:04:41.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-11"}]: dispatch 2024-03-26T02:04:41.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:40 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-11"}]: dispatch 2024-03-26T02:04:41.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3953820198' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi072-82387-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:41.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2463762264' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82643-10", "tierpool":"test-rados-api-smithi072-82643-10-cache"}]: dispatch 2024-03-26T02:04:41.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:40 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82643-10", "tierpool":"test-rados-api-smithi072-82643-10-cache"}]: dispatch 2024-03-26T02:04:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:41 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:04:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:41 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:41 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-11"}]': finished 2024-03-26T02:04:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:41 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3953820198' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi072-82387-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:41 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82643-10", "tierpool":"test-rados-api-smithi072-82643-10-cache"}]': finished 2024-03-26T02:04:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:41 smithi072 ceph-mon[30177]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T02:04:42.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:41 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-11", "mode": "writeback"}]: dispatch 2024-03-26T02:04:42.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:41 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-11", "mode": "writeback"}]: dispatch 2024-03-26T02:04:42.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:41 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:04:42.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:41 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:42.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:41 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-11"}]': finished 2024-03-26T02:04:42.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:41 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3953820198' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi072-82387-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:42.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:41 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82643-10", "tierpool":"test-rados-api-smithi072-82643-10-cache"}]': finished 2024-03-26T02:04:42.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:41 smithi072 ceph-mon[41619]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T02:04:42.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:41 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-11", "mode": "writeback"}]: dispatch 2024-03-26T02:04:42.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:41 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-11", "mode": "writeback"}]: dispatch 2024-03-26T02:04:42.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:41 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:04:42.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:41 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:42.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:41 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-11"}]': finished 2024-03-26T02:04:42.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:41 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3953820198' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi072-82387-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:42.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:41 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82643-10", "tierpool":"test-rados-api-smithi072-82643-10-cache"}]': finished 2024-03-26T02:04:42.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:41 smithi173 ceph-mon[33839]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T02:04:42.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:41 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-11", "mode": "writeback"}]: dispatch 2024-03-26T02:04:42.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:41 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-11", "mode": "writeback"}]: dispatch 2024-03-26T02:04:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:43 smithi072 ceph-mon[30177]: pgmap v133: 620 pgs: 64 unknown, 64 creating+peering, 492 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 4 op/s 2024-03-26T02:04:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:43 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:43 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:04:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:43 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-11", "mode": "writeback"}]': finished 2024-03-26T02:04:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:43 smithi072 ceph-mon[30177]: osdmap e117: 8 total, 8 up, 8 in 2024-03-26T02:04:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:43 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1218231922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi072-82404-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:43 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi072-82404-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:43 smithi072 ceph-mon[41619]: pgmap v133: 620 pgs: 64 unknown, 64 creating+peering, 492 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 4 op/s 2024-03-26T02:04:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:43 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:43 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:04:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:43 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-11", "mode": "writeback"}]': finished 2024-03-26T02:04:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:43 smithi072 ceph-mon[41619]: osdmap e117: 8 total, 8 up, 8 in 2024-03-26T02:04:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:43 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1218231922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi072-82404-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:43 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi072-82404-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:43 smithi173 ceph-mon[33839]: pgmap v133: 620 pgs: 64 unknown, 64 creating+peering, 492 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 4 op/s 2024-03-26T02:04:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:43 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:43 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:04:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:43 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-11", "mode": "writeback"}]': finished 2024-03-26T02:04:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:43 smithi173 ceph-mon[33839]: osdmap e117: 8 total, 8 up, 8 in 2024-03-26T02:04:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:43 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1218231922' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi072-82404-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:43 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi072-82404-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:43.617 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: Running main() from gmock_main.cc 2024-03-26T02:04:43.617 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-03-26T02:04:43.617 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: [----------] Global test environment set-up. 2024-03-26T02:04:43.617 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-03-26T02:04:43.617 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-03-26T02:04:43.617 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: [ OK ] LibRadosPools.PoolList (11232 ms) 2024-03-26T02:04:43.617 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-03-26T02:04:43.617 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (7123 ms) 2024-03-26T02:04:43.617 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-03-26T02:04:43.617 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3100 ms) 2024-03-26T02:04:43.618 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-03-26T02:04:43.618 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3105 ms) 2024-03-26T02:04:43.618 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-03-26T02:04:43.618 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (5384 ms) 2024-03-26T02:04:43.618 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-03-26T02:04:43.618 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5550 ms) 2024-03-26T02:04:43.618 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-03-26T02:04:43.618 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5515 ms) 2024-03-26T02:04:43.618 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-03-26T02:04:43.618 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (6000 ms) 2024-03-26T02:04:43.618 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-03-26T02:04:43.618 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (7481 ms) 2024-03-26T02:04:43.618 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: [----------] 9 tests from LibRadosPools (54491 ms total) 2024-03-26T02:04:43.618 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: 2024-03-26T02:04:43.619 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: [----------] Global test environment tear-down 2024-03-26T02:04:43.619 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (54491 ms total) 2024-03-26T02:04:43.619 INFO:tasks.workunit.client.0.smithi072.stdout: api_pool: [ PASSED ] 9 tests. 2024-03-26T02:04:44.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:44 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:44.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:44 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi072-82404-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:44.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:44 smithi072 ceph-mon[30177]: osdmap e118: 8 total, 8 up, 8 in 2024-03-26T02:04:44.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:44 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3362556373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi072-82387-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:44.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:44 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:04:44.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:44 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:04:44.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:44 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:44.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:44 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:44.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:44 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi072-82404-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:44.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:44 smithi072 ceph-mon[41619]: osdmap e118: 8 total, 8 up, 8 in 2024-03-26T02:04:44.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:44 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3362556373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi072-82387-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:44.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:44 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:04:44.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:44 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:04:44.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:44 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:44.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:44 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:44.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:44 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi072-82404-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:44.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:44 smithi173 ceph-mon[33839]: osdmap e118: 8 total, 8 up, 8 in 2024-03-26T02:04:44.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:44 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3362556373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi072-82387-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:44.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:44 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:04:44.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:44 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:04:44.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:44 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:45.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:04:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:04:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:04:45.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:45 smithi072 ceph-mon[30177]: pgmap v136: 524 pgs: 64 unknown, 460 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 4 op/s 2024-03-26T02:04:45.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:45 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3362556373' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi072-82387-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:45.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:45 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:04:45.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:45 smithi072 ceph-mon[30177]: osdmap e119: 8 total, 8 up, 8 in 2024-03-26T02:04:45.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:45 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-11"}]: dispatch 2024-03-26T02:04:45.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:45 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-11"}]: dispatch 2024-03-26T02:04:45.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:45 smithi072 ceph-mon[41619]: pgmap v136: 524 pgs: 64 unknown, 460 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 4 op/s 2024-03-26T02:04:45.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:45 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3362556373' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi072-82387-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:45.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:45 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:04:45.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:45 smithi072 ceph-mon[41619]: osdmap e119: 8 total, 8 up, 8 in 2024-03-26T02:04:45.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:45 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-11"}]: dispatch 2024-03-26T02:04:45.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:45 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-11"}]: dispatch 2024-03-26T02:04:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:45 smithi173 ceph-mon[33839]: pgmap v136: 524 pgs: 64 unknown, 460 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 258 KiB/s wr, 4 op/s 2024-03-26T02:04:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:45 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3362556373' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi072-82387-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:45 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:04:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:45 smithi173 ceph-mon[33839]: osdmap e119: 8 total, 8 up, 8 in 2024-03-26T02:04:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:45 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-11"}]: dispatch 2024-03-26T02:04:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:45 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-11"}]: dispatch 2024-03-26T02:04:46.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:46 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:46.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:46 smithi072 ceph-mon[30177]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:46.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:46 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:04:46.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:46 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-11"}]': finished 2024-03-26T02:04:46.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:46 smithi072 ceph-mon[30177]: osdmap e120: 8 total, 8 up, 8 in 2024-03-26T02:04:46.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:46 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/368199494' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi072-82404-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:46.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:46 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:46.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:46 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:46.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:46 smithi072 ceph-mon[41619]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:46.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:46 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:04:46.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:46 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-11"}]': finished 2024-03-26T02:04:46.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:46 smithi072 ceph-mon[41619]: osdmap e120: 8 total, 8 up, 8 in 2024-03-26T02:04:46.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:46 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/368199494' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi072-82404-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:46.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:46 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:46.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:46 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:46.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:46 smithi173 ceph-mon[33839]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:46.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:46 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:04:46.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:46 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-11"}]': finished 2024-03-26T02:04:46.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:46 smithi173 ceph-mon[33839]: osdmap e120: 8 total, 8 up, 8 in 2024-03-26T02:04:46.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:46 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/368199494' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi072-82404-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:46.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:46 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:47.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:47 smithi072 ceph-mon[41619]: pgmap v139: 492 pgs: 492 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-03-26T02:04:47.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:47 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/368199494' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi072-82404-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:47.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:47 smithi072 ceph-mon[41619]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T02:04:47.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:47 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2097009845' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi072-82387-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:47.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:47 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi072-82387-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:47.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:47 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi072-82728-10"}]: dispatch 2024-03-26T02:04:47.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:47 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:47.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:47 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T02:04:47.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:47 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T02:04:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:47 smithi072 ceph-mon[30177]: pgmap v139: 492 pgs: 492 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-03-26T02:04:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:47 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/368199494' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi072-82404-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:47 smithi072 ceph-mon[30177]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T02:04:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:47 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2097009845' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi072-82387-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:47 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi072-82387-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:47 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi072-82728-10"}]: dispatch 2024-03-26T02:04:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:47 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:47.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:47 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T02:04:47.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:47 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T02:04:47.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:47 smithi173 ceph-mon[33839]: pgmap v139: 492 pgs: 492 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 5.5 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-03-26T02:04:47.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:47 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/368199494' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi072-82404-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:47.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:47 smithi173 ceph-mon[33839]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T02:04:47.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:47 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2097009845' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi072-82387-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:47.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:47 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi072-82387-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:47.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:47 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi072-82728-10"}]: dispatch 2024-03-26T02:04:47.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:47 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:47.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:47 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T02:04:47.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:47 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T02:04:47.617 INFO:tasks.workunit.client.0.smithi072.stdout: io: Running main() from gmock_main.cc 2024-03-26T02:04:47.617 INFO:tasks.workunit.client.0.smithi072.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-03-26T02:04:47.617 INFO:tasks.workunit.client.0.smithi072.stdout: io: [----------] Global test environment set-up. 2024-03-26T02:04:47.617 INFO:tasks.workunit.client.0.smithi072.stdout: io: [----------] 14 tests from NeoRadosIo 2024-03-26T02:04:47.617 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-03-26T02:04:47.617 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ OK ] NeoRadosIo.Limits (12786 ms) 2024-03-26T02:04:47.617 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-03-26T02:04:47.618 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (5281 ms) 2024-03-26T02:04:47.618 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-03-26T02:04:47.618 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ OK ] NeoRadosIo.ReadOp (3093 ms) 2024-03-26T02:04:47.618 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-03-26T02:04:47.618 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ OK ] NeoRadosIo.SparseRead (3072 ms) 2024-03-26T02:04:47.618 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-03-26T02:04:47.618 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ OK ] NeoRadosIo.RoundTrip (3882 ms) 2024-03-26T02:04:47.618 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-03-26T02:04:47.618 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (4742 ms) 2024-03-26T02:04:47.618 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-03-26T02:04:47.618 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (3449 ms) 2024-03-26T02:04:47.618 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-03-26T02:04:47.618 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (3199 ms) 2024-03-26T02:04:47.618 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-03-26T02:04:47.619 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (2687 ms) 2024-03-26T02:04:47.619 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-03-26T02:04:47.619 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ OK ] NeoRadosIo.Trunc (3199 ms) 2024-03-26T02:04:47.619 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-03-26T02:04:47.619 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ OK ] NeoRadosIo.Remove (4027 ms) 2024-03-26T02:04:47.619 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-03-26T02:04:47.619 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (2786 ms) 2024-03-26T02:04:47.619 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-03-26T02:04:47.619 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ OK ] NeoRadosIo.RmXattr (3008 ms) 2024-03-26T02:04:47.619 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-03-26T02:04:47.619 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ OK ] NeoRadosIo.GetXattrs (2989 ms) 2024-03-26T02:04:47.619 INFO:tasks.workunit.client.0.smithi072.stdout: io: [----------] 14 tests from NeoRadosIo (58201 ms total) 2024-03-26T02:04:47.619 INFO:tasks.workunit.client.0.smithi072.stdout: io: 2024-03-26T02:04:47.619 INFO:tasks.workunit.client.0.smithi072.stdout: io: [----------] Global test environment tear-down 2024-03-26T02:04:47.619 INFO:tasks.workunit.client.0.smithi072.stdout: io: [==========] 14 tests from 1 test suite ran. (58201 ms total) 2024-03-26T02:04:47.620 INFO:tasks.workunit.client.0.smithi072.stdout: io: [ PASSED ] 14 tests. 2024-03-26T02:04:48.631 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: Running main() from gmock_main.cc 2024-03-26T02:04:48.632 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-03-26T02:04:48.632 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [----------] Global test environment set-up. 2024-03-26T02:04:48.632 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-03-26T02:04:48.632 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-03-26T02:04:48.632 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (12735 ms) 2024-03-26T02:04:48.632 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-03-26T02:04:48.632 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (6593 ms) 2024-03-26T02:04:48.632 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-03-26T02:04:48.632 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (3048 ms) 2024-03-26T02:04:48.632 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-03-26T02:04:48.632 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (4332 ms) 2024-03-26T02:04:48.632 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-03-26T02:04:48.632 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (3992 ms) 2024-03-26T02:04:48.633 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-03-26T02:04:48.633 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (3289 ms) 2024-03-26T02:04:48.633 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-03-26T02:04:48.633 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (3268 ms) 2024-03-26T02:04:48.633 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-03-26T02:04:48.633 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (3417 ms) 2024-03-26T02:04:48.633 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-03-26T02:04:48.633 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (3660 ms) 2024-03-26T02:04:48.633 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-03-26T02:04:48.633 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (3296 ms) 2024-03-26T02:04:48.633 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-03-26T02:04:48.633 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (2745 ms) 2024-03-26T02:04:48.633 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-03-26T02:04:48.633 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (2776 ms) 2024-03-26T02:04:48.633 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-03-26T02:04:48.633 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (2997 ms) 2024-03-26T02:04:48.634 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-03-26T02:04:48.634 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (3015 ms) 2024-03-26T02:04:48.634 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (59163 ms total) 2024-03-26T02:04:48.634 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: 2024-03-26T02:04:48.634 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [----------] Global test environment tear-down 2024-03-26T02:04:48.634 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (59163 ms total) 2024-03-26T02:04:48.634 INFO:tasks.workunit.client.0.smithi072.stdout: read_operations: [ PASSED ] 14 tests. 2024-03-26T02:04:48.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:48 smithi072 ceph-mon[41619]: pgmap v142: 516 pgs: 128 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T02:04:48.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:48 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi072-82387-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:48.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:48 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi072-82728-10"}]': finished 2024-03-26T02:04:48.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:48 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pg_num","val":"11"}]': finished 2024-03-26T02:04:48.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:48 smithi072 ceph-mon[41619]: osdmap e122: 8 total, 8 up, 8 in 2024-03-26T02:04:48.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:48 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:48.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:48 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi072-82728-10"}]: dispatch 2024-03-26T02:04:48.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:48 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:48.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:48 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:48.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:48 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:48.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:48 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T02:04:48.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:48 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T02:04:48.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:48 smithi072 ceph-mon[30177]: pgmap v142: 516 pgs: 128 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T02:04:48.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:48 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi072-82387-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:48.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:48 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi072-82728-10"}]': finished 2024-03-26T02:04:48.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:48 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pg_num","val":"11"}]': finished 2024-03-26T02:04:48.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:48 smithi072 ceph-mon[30177]: osdmap e122: 8 total, 8 up, 8 in 2024-03-26T02:04:48.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:48 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:48.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:48 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi072-82728-10"}]: dispatch 2024-03-26T02:04:48.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:48 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:48.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:48 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:48.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:48 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:48.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:48 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T02:04:48.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:48 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T02:04:49.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:48 smithi173 ceph-mon[33839]: pgmap v142: 516 pgs: 128 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T02:04:49.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:48 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi072-82387-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:49.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:48 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi072-82728-10"}]': finished 2024-03-26T02:04:49.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:48 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pg_num","val":"11"}]': finished 2024-03-26T02:04:49.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:48 smithi173 ceph-mon[33839]: osdmap e122: 8 total, 8 up, 8 in 2024-03-26T02:04:49.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:48 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:49.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:48 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi072-82728-10"}]: dispatch 2024-03-26T02:04:49.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:48 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:49.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:48 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:49.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:48 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:49.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:48 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T02:04:49.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:48 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T02:04:49.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi072-82728-10"}]': finished 2024-03-26T02:04:49.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:49.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T02:04:49.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[30177]: osdmap e123: 8 total, 8 up, 8 in 2024-03-26T02:04:49.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:49.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3322642493' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi072-82404-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:49.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi072-82404-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:49.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2079532164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:04:49.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:04:49.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2079532164' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:04:49.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:04:49.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2079532164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:49.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:49.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi072-82404-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:49.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:49.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[30177]: osdmap e124: 8 total, 8 up, 8 in 2024-03-26T02:04:49.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2079532164' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi072-82728-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:04:49.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi072-82728-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:04:49.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi072-82728-10"}]': finished 2024-03-26T02:04:49.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:49.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T02:04:49.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[41619]: osdmap e123: 8 total, 8 up, 8 in 2024-03-26T02:04:49.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:49.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3322642493' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi072-82404-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:49.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi072-82404-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:49.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2079532164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:04:49.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:04:49.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2079532164' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:04:49.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:04:49.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2079532164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:49.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:49.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi072-82404-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:49.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:49.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[41619]: osdmap e124: 8 total, 8 up, 8 in 2024-03-26T02:04:49.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2079532164' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi072-82728-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:04:49.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi072-82728-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:04:49.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2359670207' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi072-82387-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:49.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:49.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi072-82387-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:49.973 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2359670207' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi072-82387-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:49.973 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:49.973 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:49 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi072-82387-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:50.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:49 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2324514372' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi072-82728-10"}]': finished 2024-03-26T02:04:50.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:49 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:50.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:49 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T02:04:50.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:49 smithi173 ceph-mon[33839]: osdmap e123: 8 total, 8 up, 8 in 2024-03-26T02:04:50.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:49 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:50.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:49 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3322642493' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi072-82404-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:50.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:49 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi072-82404-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:50.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:49 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2079532164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:04:50.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:49 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:04:50.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:49 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2079532164' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:04:50.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:49 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:04:50.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:49 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2079532164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:50.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:49 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:04:50.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:49 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi072-82404-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:50.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:49 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:04:50.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:49 smithi173 ceph-mon[33839]: osdmap e124: 8 total, 8 up, 8 in 2024-03-26T02:04:50.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:49 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2079532164' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi072-82728-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:04:50.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:49 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi072-82728-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:04:50.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:49 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2359670207' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi072-82387-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:50.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:49 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:50.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:49 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi072-82387-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:50.862 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:04:50 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:04:51.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:50 smithi072 ceph-mon[30177]: pgmap v145: 452 pgs: 64 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-26T02:04:51.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:50 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:51.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:50 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi072-82387-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:51.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:50 smithi072 ceph-mon[41619]: pgmap v145: 452 pgs: 64 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-26T02:04:51.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:50 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:51.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:50 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi072-82387-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:51.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:50 smithi173 ceph-mon[33839]: pgmap v145: 452 pgs: 64 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail 2024-03-26T02:04:51.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:50 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:51.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:50 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi072-82387-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:52.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:51 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:04:52.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:51 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:52.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:51 smithi072 ceph-mon[30177]: osdmap e125: 8 total, 8 up, 8 in 2024-03-26T02:04:52.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:51 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:04:52.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:51 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi072-82728-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]': finished 2024-03-26T02:04:52.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:51 smithi072 ceph-mon[30177]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T02:04:52.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:51 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/254936014' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi072-82404-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:52.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:51 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:52.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:51 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi072-82404-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:52.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:51 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:04:52.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:51 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:52.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:51 smithi072 ceph-mon[41619]: osdmap e125: 8 total, 8 up, 8 in 2024-03-26T02:04:52.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:51 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:04:52.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:51 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi072-82728-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]': finished 2024-03-26T02:04:52.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:51 smithi072 ceph-mon[41619]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T02:04:52.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:51 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/254936014' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi072-82404-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:52.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:51 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:52.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:51 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi072-82404-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:51 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:04:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:51 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:51 smithi173 ceph-mon[33839]: osdmap e125: 8 total, 8 up, 8 in 2024-03-26T02:04:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:51 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:04:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:51 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi072-82728-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]': finished 2024-03-26T02:04:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:51 smithi173 ceph-mon[33839]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T02:04:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:51 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/254936014' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi072-82404-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:51 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:51 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi072-82404-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:53.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:52 smithi072 ceph-mon[41619]: pgmap v148: 452 pgs: 32 creating+peering, 420 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 51 op/s 2024-03-26T02:04:53.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:52 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi072-82404-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:53.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:52 smithi072 ceph-mon[41619]: osdmap e127: 8 total, 8 up, 8 in 2024-03-26T02:04:53.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:52 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3478779932' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi072-82387-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:53.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:52 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:53.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:52 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi072-82387-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:53.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:52 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:53.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:52 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:53.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:52 smithi072 ceph-mon[30177]: pgmap v148: 452 pgs: 32 creating+peering, 420 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 51 op/s 2024-03-26T02:04:53.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:52 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi072-82404-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:53.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:52 smithi072 ceph-mon[30177]: osdmap e127: 8 total, 8 up, 8 in 2024-03-26T02:04:53.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:52 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3478779932' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi072-82387-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:53.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:52 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:53.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:52 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi072-82387-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:53.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:52 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:53.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:52 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:52 smithi173 ceph-mon[33839]: pgmap v148: 452 pgs: 32 creating+peering, 420 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 51 op/s 2024-03-26T02:04:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:52 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi072-82404-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:52 smithi173 ceph-mon[33839]: osdmap e127: 8 total, 8 up, 8 in 2024-03-26T02:04:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:52 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3478779932' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi072-82387-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:52 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:52 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi072-82387-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:52 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:52 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:04:54.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:54 smithi072 ceph-mon[30177]: pgmap v151: 492 pgs: 72 unknown, 420 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 51 op/s 2024-03-26T02:04:54.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:54 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi072-82387-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:54.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:54 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:04:54.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:54 smithi072 ceph-mon[30177]: osdmap e128: 8 total, 8 up, 8 in 2024-03-26T02:04:54.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:54 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-13"}]: dispatch 2024-03-26T02:04:54.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:54 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-13"}]: dispatch 2024-03-26T02:04:54.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:54 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:54.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:54 smithi072 ceph-mon[41619]: pgmap v151: 492 pgs: 72 unknown, 420 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 51 op/s 2024-03-26T02:04:54.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:54 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi072-82387-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:54.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:54 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:04:54.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:54 smithi072 ceph-mon[41619]: osdmap e128: 8 total, 8 up, 8 in 2024-03-26T02:04:54.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:54 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-13"}]: dispatch 2024-03-26T02:04:54.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:54 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-13"}]: dispatch 2024-03-26T02:04:54.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:54 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:55.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:54 smithi173 ceph-mon[33839]: pgmap v151: 492 pgs: 72 unknown, 420 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 50 KiB/s wr, 51 op/s 2024-03-26T02:04:55.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:54 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi072-82387-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:55.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:54 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:04:55.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:54 smithi173 ceph-mon[33839]: osdmap e128: 8 total, 8 up, 8 in 2024-03-26T02:04:55.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:54 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-13"}]: dispatch 2024-03-26T02:04:55.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:54 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-13"}]: dispatch 2024-03-26T02:04:55.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:54 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:55.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:04:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:04:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:04:55.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-13"}]': finished 2024-03-26T02:04:55.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[30177]: osdmap e129: 8 total, 8 up, 8 in 2024-03-26T02:04:55.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-13", "mode": "writeback"}]: dispatch 2024-03-26T02:04:55.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1142046619' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi072-82404-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:55.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-13", "mode": "writeback"}]: dispatch 2024-03-26T02:04:55.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi072-82404-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:55.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:55.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[30177]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:55.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:04:55.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-13", "mode": "writeback"}]': finished 2024-03-26T02:04:55.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi072-82404-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:55.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[30177]: osdmap e130: 8 total, 8 up, 8 in 2024-03-26T02:04:55.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2438720258' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi072-82387-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:55.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi072-82387-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:55.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-13"}]': finished 2024-03-26T02:04:55.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[41619]: osdmap e129: 8 total, 8 up, 8 in 2024-03-26T02:04:55.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-13", "mode": "writeback"}]: dispatch 2024-03-26T02:04:55.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1142046619' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi072-82404-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:55.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-13", "mode": "writeback"}]: dispatch 2024-03-26T02:04:55.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi072-82404-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:55.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:55.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[41619]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:55.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:04:55.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-13", "mode": "writeback"}]': finished 2024-03-26T02:04:55.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi072-82404-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:55.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[41619]: osdmap e130: 8 total, 8 up, 8 in 2024-03-26T02:04:55.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2438720258' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi072-82387-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:55.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:55 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi072-82387-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:56.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:55 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-13"}]': finished 2024-03-26T02:04:56.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:55 smithi173 ceph-mon[33839]: osdmap e129: 8 total, 8 up, 8 in 2024-03-26T02:04:56.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:55 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-13", "mode": "writeback"}]: dispatch 2024-03-26T02:04:56.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:55 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1142046619' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi072-82404-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:56.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:55 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-13", "mode": "writeback"}]: dispatch 2024-03-26T02:04:56.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:55 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi072-82404-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:56.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:55 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:56.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:55 smithi173 ceph-mon[33839]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:04:56.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:55 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:04:56.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:55 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-13", "mode": "writeback"}]': finished 2024-03-26T02:04:56.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:55 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi072-82404-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:56.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:55 smithi173 ceph-mon[33839]: osdmap e130: 8 total, 8 up, 8 in 2024-03-26T02:04:56.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:55 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2438720258' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi072-82387-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:56.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:55 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi072-82387-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:56.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[41619]: pgmap v154: 460 pgs: 32 creating+peering, 428 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 79 KiB/s wr, 76 op/s 2024-03-26T02:04:56.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:56.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.0"}]: dispatch 2024-03-26T02:04:56.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.1"}]: dispatch 2024-03-26T02:04:56.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.2"}]: dispatch 2024-03-26T02:04:56.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.3"}]: dispatch 2024-03-26T02:04:56.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.4"}]: dispatch 2024-03-26T02:04:56.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.5"}]: dispatch 2024-03-26T02:04:56.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.6"}]: dispatch 2024-03-26T02:04:56.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.7"}]: dispatch 2024-03-26T02:04:56.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.8"}]: dispatch 2024-03-26T02:04:56.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.9"}]: dispatch 2024-03-26T02:04:56.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi072-82387-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:56.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[41619]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T02:04:56.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:56.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[30177]: pgmap v154: 460 pgs: 32 creating+peering, 428 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 79 KiB/s wr, 76 op/s 2024-03-26T02:04:56.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:56.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.0"}]: dispatch 2024-03-26T02:04:56.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.1"}]: dispatch 2024-03-26T02:04:56.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.2"}]: dispatch 2024-03-26T02:04:56.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.3"}]: dispatch 2024-03-26T02:04:56.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.4"}]: dispatch 2024-03-26T02:04:56.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.5"}]: dispatch 2024-03-26T02:04:56.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.6"}]: dispatch 2024-03-26T02:04:56.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.7"}]: dispatch 2024-03-26T02:04:56.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.8"}]: dispatch 2024-03-26T02:04:56.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.9"}]: dispatch 2024-03-26T02:04:56.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi072-82387-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:56.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[30177]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T02:04:56.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:56 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:57.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:56 smithi173 ceph-mon[33839]: pgmap v154: 460 pgs: 32 creating+peering, 428 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 79 KiB/s wr, 76 op/s 2024-03-26T02:04:57.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:56 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:57.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:56 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.0"}]: dispatch 2024-03-26T02:04:57.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:56 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.1"}]: dispatch 2024-03-26T02:04:57.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:56 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.2"}]: dispatch 2024-03-26T02:04:57.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:56 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.3"}]: dispatch 2024-03-26T02:04:57.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:56 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.4"}]: dispatch 2024-03-26T02:04:57.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:56 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.5"}]: dispatch 2024-03-26T02:04:57.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:56 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.6"}]: dispatch 2024-03-26T02:04:57.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:56 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.7"}]: dispatch 2024-03-26T02:04:57.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:56 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.8"}]: dispatch 2024-03-26T02:04:57.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:56 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "165.9"}]: dispatch 2024-03-26T02:04:57.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:56 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi072-82387-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:57.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:56 smithi173 ceph-mon[33839]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T02:04:57.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:56 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:57.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.0"}]: dispatch 2024-03-26T02:04:57.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.1"}]: dispatch 2024-03-26T02:04:57.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.2"}]: dispatch 2024-03-26T02:04:57.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.3"}]: dispatch 2024-03-26T02:04:57.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.4"}]: dispatch 2024-03-26T02:04:57.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.5"}]: dispatch 2024-03-26T02:04:57.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.6"}]: dispatch 2024-03-26T02:04:57.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.7"}]: dispatch 2024-03-26T02:04:57.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.8"}]: dispatch 2024-03-26T02:04:57.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.9"}]: dispatch 2024-03-26T02:04:57.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[41619]: 165.6 deep-scrub starts 2024-03-26T02:04:57.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[41619]: 165.6 deep-scrub ok 2024-03-26T02:04:57.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[41619]: 165.3 scrub starts 2024-03-26T02:04:57.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[41619]: 165.3 scrub ok 2024-03-26T02:04:57.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[41619]: 165.8 scrub starts 2024-03-26T02:04:57.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[41619]: 165.8 scrub ok 2024-03-26T02:04:57.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:57.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.0"}]: dispatch 2024-03-26T02:04:57.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.1"}]: dispatch 2024-03-26T02:04:57.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.2"}]: dispatch 2024-03-26T02:04:57.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.3"}]: dispatch 2024-03-26T02:04:57.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.4"}]: dispatch 2024-03-26T02:04:57.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.5"}]: dispatch 2024-03-26T02:04:57.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.6"}]: dispatch 2024-03-26T02:04:57.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.7"}]: dispatch 2024-03-26T02:04:57.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.8"}]: dispatch 2024-03-26T02:04:57.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.9"}]: dispatch 2024-03-26T02:04:57.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[30177]: 165.6 deep-scrub starts 2024-03-26T02:04:57.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[30177]: 165.6 deep-scrub ok 2024-03-26T02:04:57.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[30177]: 165.3 scrub starts 2024-03-26T02:04:57.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[30177]: 165.3 scrub ok 2024-03-26T02:04:57.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[30177]: 165.8 scrub starts 2024-03-26T02:04:57.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[30177]: 165.8 scrub ok 2024-03-26T02:04:57.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:57 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:58.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:57 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.0"}]: dispatch 2024-03-26T02:04:58.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:57 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.1"}]: dispatch 2024-03-26T02:04:58.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:57 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.2"}]: dispatch 2024-03-26T02:04:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:57 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.3"}]: dispatch 2024-03-26T02:04:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:57 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.4"}]: dispatch 2024-03-26T02:04:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:57 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.5"}]: dispatch 2024-03-26T02:04:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:57 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.6"}]: dispatch 2024-03-26T02:04:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:57 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.7"}]: dispatch 2024-03-26T02:04:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:57 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.8"}]: dispatch 2024-03-26T02:04:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:57 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "165.9"}]: dispatch 2024-03-26T02:04:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:57 smithi173 ceph-mon[33839]: 165.6 deep-scrub starts 2024-03-26T02:04:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:57 smithi173 ceph-mon[33839]: 165.6 deep-scrub ok 2024-03-26T02:04:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:57 smithi173 ceph-mon[33839]: 165.3 scrub starts 2024-03-26T02:04:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:57 smithi173 ceph-mon[33839]: 165.3 scrub ok 2024-03-26T02:04:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:57 smithi173 ceph-mon[33839]: 165.8 scrub starts 2024-03-26T02:04:58.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:57 smithi173 ceph-mon[33839]: 165.8 scrub ok 2024-03-26T02:04:58.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:57 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:58.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[30177]: 165.2 scrub starts 2024-03-26T02:04:58.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[30177]: 165.2 scrub ok 2024-03-26T02:04:58.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[30177]: 165.7 scrub starts 2024-03-26T02:04:58.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[30177]: 165.7 scrub ok 2024-03-26T02:04:58.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[30177]: pgmap v157: 460 pgs: 32 unknown, 428 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 79 KiB/s wr, 76 op/s 2024-03-26T02:04:58.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[30177]: 165.0 scrub starts 2024-03-26T02:04:58.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[30177]: 165.0 scrub ok 2024-03-26T02:04:58.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[30177]: osdmap e132: 8 total, 8 up, 8 in 2024-03-26T02:04:58.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/548456037' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi072-82404-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:58.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[30177]: 165.4 scrub starts 2024-03-26T02:04:58.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[30177]: 165.4 scrub ok 2024-03-26T02:04:58.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[30177]: 165.1 deep-scrub starts 2024-03-26T02:04:58.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[30177]: 165.1 deep-scrub ok 2024-03-26T02:04:58.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:58.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[41619]: 165.2 scrub starts 2024-03-26T02:04:58.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[41619]: 165.2 scrub ok 2024-03-26T02:04:58.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[41619]: 165.7 scrub starts 2024-03-26T02:04:58.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[41619]: 165.7 scrub ok 2024-03-26T02:04:58.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[41619]: pgmap v157: 460 pgs: 32 unknown, 428 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 79 KiB/s wr, 76 op/s 2024-03-26T02:04:58.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[41619]: 165.0 scrub starts 2024-03-26T02:04:58.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[41619]: 165.0 scrub ok 2024-03-26T02:04:58.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[41619]: osdmap e132: 8 total, 8 up, 8 in 2024-03-26T02:04:58.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/548456037' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi072-82404-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:58.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[41619]: 165.4 scrub starts 2024-03-26T02:04:58.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[41619]: 165.4 scrub ok 2024-03-26T02:04:58.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[41619]: 165.1 deep-scrub starts 2024-03-26T02:04:58.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[41619]: 165.1 deep-scrub ok 2024-03-26T02:04:58.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:59.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:58 smithi173 ceph-mon[33839]: 165.2 scrub starts 2024-03-26T02:04:59.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:58 smithi173 ceph-mon[33839]: 165.2 scrub ok 2024-03-26T02:04:59.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:58 smithi173 ceph-mon[33839]: 165.7 scrub starts 2024-03-26T02:04:59.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:58 smithi173 ceph-mon[33839]: 165.7 scrub ok 2024-03-26T02:04:59.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:58 smithi173 ceph-mon[33839]: pgmap v157: 460 pgs: 32 unknown, 428 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 79 KiB/s wr, 76 op/s 2024-03-26T02:04:59.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:58 smithi173 ceph-mon[33839]: 165.0 scrub starts 2024-03-26T02:04:59.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:58 smithi173 ceph-mon[33839]: 165.0 scrub ok 2024-03-26T02:04:59.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:58 smithi173 ceph-mon[33839]: osdmap e132: 8 total, 8 up, 8 in 2024-03-26T02:04:59.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/548456037' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi072-82404-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:59.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:58 smithi173 ceph-mon[33839]: 165.4 scrub starts 2024-03-26T02:04:59.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:58 smithi173 ceph-mon[33839]: 165.4 scrub ok 2024-03-26T02:04:59.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:58 smithi173 ceph-mon[33839]: 165.1 deep-scrub starts 2024-03-26T02:04:59.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:58 smithi173 ceph-mon[33839]: 165.1 deep-scrub ok 2024-03-26T02:04:59.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:59.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:59 smithi072 ceph-mon[30177]: 165.5 scrub starts 2024-03-26T02:04:59.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:59 smithi072 ceph-mon[30177]: 165.5 scrub ok 2024-03-26T02:04:59.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:59 smithi072 ceph-mon[30177]: 165.9 scrub starts 2024-03-26T02:04:59.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:59 smithi072 ceph-mon[30177]: 165.9 scrub ok 2024-03-26T02:04:59.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:59 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/548456037' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi072-82404-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:59.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:59 smithi072 ceph-mon[30177]: osdmap e133: 8 total, 8 up, 8 in 2024-03-26T02:04:59.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:59 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4217867721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi072-82387-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:59.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:59 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi072-82387-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:59.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:04:59 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:04:59.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:59 smithi072 ceph-mon[41619]: 165.5 scrub starts 2024-03-26T02:04:59.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:59 smithi072 ceph-mon[41619]: 165.5 scrub ok 2024-03-26T02:04:59.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:59 smithi072 ceph-mon[41619]: 165.9 scrub starts 2024-03-26T02:04:59.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:59 smithi072 ceph-mon[41619]: 165.9 scrub ok 2024-03-26T02:04:59.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:59 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/548456037' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi072-82404-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:04:59.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:59 smithi072 ceph-mon[41619]: osdmap e133: 8 total, 8 up, 8 in 2024-03-26T02:04:59.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:59 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4217867721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi072-82387-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:59.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:59 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi072-82387-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:04:59.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:04:59 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:00.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:59 smithi173 ceph-mon[33839]: 165.5 scrub starts 2024-03-26T02:05:00.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:59 smithi173 ceph-mon[33839]: 165.5 scrub ok 2024-03-26T02:05:00.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:59 smithi173 ceph-mon[33839]: 165.9 scrub starts 2024-03-26T02:05:00.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:59 smithi173 ceph-mon[33839]: 165.9 scrub ok 2024-03-26T02:05:00.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:59 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/548456037' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi072-82404-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:00.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:59 smithi173 ceph-mon[33839]: osdmap e133: 8 total, 8 up, 8 in 2024-03-26T02:05:00.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:59 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4217867721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi072-82387-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:00.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:59 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi072-82387-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:00.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:04:59 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:00.897 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:05:00 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:05:01.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:00 smithi072 ceph-mon[30177]: pgmap v160: 460 pgs: 64 unknown, 396 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T02:05:01.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:00 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi072-82387-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:01.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:00 smithi072 ceph-mon[30177]: osdmap e134: 8 total, 8 up, 8 in 2024-03-26T02:05:01.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/75633135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82705-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:01.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/75633135' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82705-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:01.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:00 smithi072 ceph-mon[30177]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T02:05:01.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2079532164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:05:01.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:00 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:05:01.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:01.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/814518717' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi072-82404-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:01.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:00 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi072-82404-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:01.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:00 smithi072 ceph-mon[41619]: pgmap v160: 460 pgs: 64 unknown, 396 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T02:05:01.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:00 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi072-82387-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:01.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:00 smithi072 ceph-mon[41619]: osdmap e134: 8 total, 8 up, 8 in 2024-03-26T02:05:01.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/75633135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82705-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:01.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/75633135' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82705-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:01.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:00 smithi072 ceph-mon[41619]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T02:05:01.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2079532164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:05:01.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:00 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:05:01.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:01.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/814518717' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi072-82404-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:01.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:00 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi072-82404-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:01.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:00 smithi173 ceph-mon[33839]: pgmap v160: 460 pgs: 64 unknown, 396 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T02:05:01.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:00 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi072-82387-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:01.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:00 smithi173 ceph-mon[33839]: osdmap e134: 8 total, 8 up, 8 in 2024-03-26T02:05:01.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/75633135' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82705-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:01.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/75633135' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82705-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:01.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:00 smithi173 ceph-mon[33839]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T02:05:01.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2079532164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:05:01.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:00 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:05:01.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:01.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/814518717' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi072-82404-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:01.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:00 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi072-82404-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:01 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:05:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:01 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]': finished 2024-03-26T02:05:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:01 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi072-82404-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:01 smithi072 ceph-mon[30177]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T02:05:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1763354123' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi072-82387-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:01 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi072-82387-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2079532164' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:05:02.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:01 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:05:02.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:01 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:05:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:01 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]': finished 2024-03-26T02:05:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:01 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi072-82404-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:01 smithi072 ceph-mon[41619]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T02:05:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1763354123' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi072-82387-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:01 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi072-82387-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2079532164' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:05:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:01 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:05:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:01 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:05:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:01 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]': finished 2024-03-26T02:05:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:01 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi072-82404-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:01 smithi173 ceph-mon[33839]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T02:05:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1763354123' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi072-82387-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:01 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi072-82387-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:02.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2079532164' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:05:02.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:01 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]: dispatch 2024-03-26T02:05:02.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:02.621 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: Running main() from gmock_main.cc 2024-03-26T02:05:02.621 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-03-26T02:05:02.621 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [----------] Global test environment set-up. 2024-03-26T02:05:02.621 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-03-26T02:05:02.621 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-03-26T02:05:02.621 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (5625 ms) 2024-03-26T02:05:02.621 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-03-26T02:05:02.621 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (4334 ms) 2024-03-26T02:05:02.621 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-03-26T02:05:02.622 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (3049 ms) 2024-03-26T02:05:02.622 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-03-26T02:05:02.622 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (1786 ms) 2024-03-26T02:05:02.622 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (14794 ms total) 2024-03-26T02:05:02.622 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: 2024-03-26T02:05:02.622 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-03-26T02:05:02.622 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-03-26T02:05:02.622 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4426 ms) 2024-03-26T02:05:02.622 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-03-26T02:05:02.622 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4475 ms) 2024-03-26T02:05:02.622 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-03-26T02:05:02.622 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (5223 ms) 2024-03-26T02:05:02.622 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (14125 ms total) 2024-03-26T02:05:02.623 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: 2024-03-26T02:05:02.623 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-03-26T02:05:02.623 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-03-26T02:05:02.623 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (3023 ms) 2024-03-26T02:05:02.623 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-03-26T02:05:02.623 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (1775 ms) 2024-03-26T02:05:02.623 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-03-26T02:05:02.623 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2005 ms) 2024-03-26T02:05:02.623 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-03-26T02:05:02.623 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (1994 ms) 2024-03-26T02:05:02.623 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (8797 ms total) 2024-03-26T02:05:02.623 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: 2024-03-26T02:05:02.623 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-03-26T02:05:02.623 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-03-26T02:05:02.624 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4042 ms) 2024-03-26T02:05:02.624 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-03-26T02:05:02.624 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4062 ms) 2024-03-26T02:05:02.624 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8104 ms total) 2024-03-26T02:05:02.624 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: 2024-03-26T02:05:02.624 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [----------] Global test environment tear-down 2024-03-26T02:05:02.624 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (73464 ms total) 2024-03-26T02:05:02.624 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-03-26T02:05:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:02 smithi072 ceph-mon[30177]: pgmap v163: 452 pgs: 32 creating+peering, 32 unknown, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 54 KiB/s rd, 73 op/s 2024-03-26T02:05:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:02 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T02:05:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:02 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:02 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi072-82387-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:02 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]': finished 2024-03-26T02:05:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:02 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:02 smithi072 ceph-mon[30177]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T02:05:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "app2"}]: dispatch 2024-03-26T02:05:03.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:03.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:02 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:03.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:03.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:02 smithi072 ceph-mon[41619]: pgmap v163: 452 pgs: 32 creating+peering, 32 unknown, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 54 KiB/s rd, 73 op/s 2024-03-26T02:05:03.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:02 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:03.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T02:05:03.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:03.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:02 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:03.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:02 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi072-82387-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:03.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:02 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]': finished 2024-03-26T02:05:03.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:02 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:03.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:02 smithi072 ceph-mon[41619]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T02:05:03.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "app2"}]: dispatch 2024-03-26T02:05:03.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:03.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:02 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:03.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:02 smithi173 ceph-mon[33839]: pgmap v163: 452 pgs: 32 creating+peering, 32 unknown, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 54 KiB/s rd, 73 op/s 2024-03-26T02:05:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:02 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T02:05:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:02 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:02 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi072-82387-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:02 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi072-82728-15"}]': finished 2024-03-26T02:05:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:02 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:02 smithi173 ceph-mon[33839]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T02:05:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "app2"}]: dispatch 2024-03-26T02:05:03.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:03.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:02 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:03.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:04.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[30177]: pgmap v166: 452 pgs: 32 creating+peering, 32 unknown, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 76 op/s 2024-03-26T02:05:04.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:04.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[30177]: osdmap e138: 8 total, 8 up, 8 in 2024-03-26T02:05:04.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/983314482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi072-82404-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:04.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi072-82627-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-26T02:05:04.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi072-82404-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:04.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T02:05:04.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T02:05:04.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:04.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4134971250' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:04.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:04.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4134971250' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:04.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:04.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4134971250' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi072-82705-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:04.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi072-82705-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:04.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[41619]: pgmap v166: 452 pgs: 32 creating+peering, 32 unknown, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 76 op/s 2024-03-26T02:05:04.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:04.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[41619]: osdmap e138: 8 total, 8 up, 8 in 2024-03-26T02:05:04.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/983314482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi072-82404-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:04.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi072-82627-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-26T02:05:04.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi072-82404-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:04.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T02:05:04.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T02:05:04.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:04.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4134971250' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:04.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:04.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4134971250' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:04.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:04.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4134971250' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi072-82705-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:04.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi072-82705-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:05.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:04 smithi173 ceph-mon[33839]: pgmap v166: 452 pgs: 32 creating+peering, 32 unknown, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 76 op/s 2024-03-26T02:05:05.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:04 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi072-82627-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:05.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:04 smithi173 ceph-mon[33839]: osdmap e138: 8 total, 8 up, 8 in 2024-03-26T02:05:05.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:04 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/983314482' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi072-82404-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:05.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:04 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi072-82627-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-26T02:05:05.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:04 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi072-82404-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:05.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:04 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T02:05:05.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:04 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T02:05:05.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:04 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:05.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:04 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4134971250' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:05.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:04 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:05.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:04 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4134971250' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:05.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:04 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:05.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:04 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4134971250' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi072-82705-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:05.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:04 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi072-82705-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:05.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:05:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:05:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:05:05.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi072-82404-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:05.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T02:05:05.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi072-82705-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:05.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:05 smithi072 ceph-mon[30177]: osdmap e139: 8 total, 8 up, 8 in 2024-03-26T02:05:05.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T02:05:05.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T02:05:05.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4134971250' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi072-82705-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:05.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3565533047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi072-82387-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:05.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi072-82705-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:05.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:05.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi072-82404-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:05.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T02:05:05.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi072-82705-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:05.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:05 smithi072 ceph-mon[41619]: osdmap e139: 8 total, 8 up, 8 in 2024-03-26T02:05:05.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T02:05:05.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T02:05:05.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4134971250' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi072-82705-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:05.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3565533047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi072-82387-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:05.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi072-82705-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:05.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:06.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi072-82404-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:06.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T02:05:06.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi072-82705-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:06.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:05 smithi173 ceph-mon[33839]: osdmap e139: 8 total, 8 up, 8 in 2024-03-26T02:05:06.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T02:05:06.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T02:05:06.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4134971250' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi072-82705-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:06.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3565533047' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi072-82387-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:06.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi072-82705-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:06.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:06.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:06 smithi072 ceph-mon[30177]: pgmap v169: 452 pgs: 32 creating+peering, 420 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T02:05:06.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:06 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T02:05:06.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3565533047' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi072-82387-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:06.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T02:05:06.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:06.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:06 smithi072 ceph-mon[30177]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T02:05:06.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:06 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T02:05:06.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:06 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:05:06.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:06 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:05:06.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:06 smithi072 ceph-mon[41619]: pgmap v169: 452 pgs: 32 creating+peering, 420 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T02:05:06.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:06 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T02:05:06.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3565533047' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi072-82387-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:06.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T02:05:06.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:06.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:06 smithi072 ceph-mon[41619]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T02:05:06.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:06 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T02:05:06.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:06 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:05:06.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:06 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:05:07.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:06 smithi173 ceph-mon[33839]: pgmap v169: 452 pgs: 32 creating+peering, 420 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T02:05:07.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:06 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T02:05:07.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3565533047' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi072-82387-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:07.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1587004591' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T02:05:07.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:07.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:06 smithi173 ceph-mon[33839]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T02:05:07.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:06 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T02:05:07.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:06 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:05:07.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:06 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:05:07.657 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-03-26T02:05:07.657 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-03-26T02:05:07.657 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-03-26T02:05:07.657 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-03-26T02:05:07.658 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-03-26T02:05:07.658 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-26T02:05:07.658 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: 2024-03-26T02:05:07.658 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-03-26T02:05:07.658 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: seed 82627 2024-03-26T02:05:07.658 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-03-26T02:05:07.658 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (0 ms) 2024-03-26T02:05:07.658 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-03-26T02:05:07.658 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (305 ms) 2024-03-26T02:05:07.658 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-03-26T02:05:07.658 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (29 ms) 2024-03-26T02:05:07.658 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-03-26T02:05:07.658 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (10 ms) 2024-03-26T02:05:07.658 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-03-26T02:05:07.658 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (8 ms) 2024-03-26T02:05:07.659 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-03-26T02:05:07.659 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (2 ms) 2024-03-26T02:05:07.659 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-03-26T02:05:07.659 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (2 ms) 2024-03-26T02:05:07.659 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-03-26T02:05:07.659 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (6 ms) 2024-03-26T02:05:07.659 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-03-26T02:05:07.659 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (2 ms) 2024-03-26T02:05:07.659 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-03-26T02:05:07.659 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (10 ms) 2024-03-26T02:05:07.659 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-03-26T02:05:07.659 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (2 ms) 2024-03-26T02:05:07.659 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-03-26T02:05:07.659 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (7 ms) 2024-03-26T02:05:07.659 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-03-26T02:05:07.660 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (8 ms) 2024-03-26T02:05:07.660 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-03-26T02:05:07.660 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: osd_max_attr_size = 0 2024-03-26T02:05:07.660 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-03-26T02:05:07.660 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (1558 ms) 2024-03-26T02:05:07.660 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-03-26T02:05:07.660 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (364 ms) 2024-03-26T02:05:07.660 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-03-26T02:05:07.660 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-03-26T02:05:07.660 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: done waiting, doing copies 2024-03-26T02:05:07.660 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: waiting for final deep scrubs... 2024-03-26T02:05:07.660 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: done waiting 2024-03-26T02:05:07.660 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (61032 ms) 2024-03-26T02:05:07.660 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-03-26T02:05:07.660 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (5 ms) 2024-03-26T02:05:07.661 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-03-26T02:05:07.661 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (2 ms) 2024-03-26T02:05:07.661 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-03-26T02:05:07.661 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4468 ms) 2024-03-26T02:05:07.661 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-03-26T02:05:07.661 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-03-26T02:05:07.661 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-03-26T02:05:07.661 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-03-26T02:05:07.662 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-03-26T02:05:07.662 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-03-26T02:05:07.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:07 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi072-82705-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi072-82705-16"}]': finished 2024-03-26T02:05:07.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:07 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key1"}]': finished 2024-03-26T02:05:07.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:07 smithi072 ceph-mon[41619]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T02:05:07.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/975399653' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi072-82404-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:07.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:07 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi072-82404-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:07.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:07.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:07 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi072-82705-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi072-82705-16"}]': finished 2024-03-26T02:05:07.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:07 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key1"}]': finished 2024-03-26T02:05:07.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:07 smithi072 ceph-mon[30177]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T02:05:07.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/975399653' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi072-82404-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:07.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:07 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi072-82404-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:07.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:08.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:07 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi072-82705-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi072-82705-16"}]': finished 2024-03-26T02:05:08.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:07 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi072-82627-1","app":"app1","key":"key1"}]': finished 2024-03-26T02:05:08.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:07 smithi173 ceph-mon[33839]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T02:05:08.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/975399653' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi072-82404-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:08.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:07 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi072-82404-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:08.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:08.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:08 smithi072 ceph-mon[30177]: pgmap v172: 428 pgs: 40 unknown, 388 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T02:05:08.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:08 smithi072 ceph-mon[30177]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:08.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi072-82404-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:08.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:08 smithi072 ceph-mon[30177]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T02:05:08.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1233630792' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi072-82387-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:08.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:08.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82627-24"}]: dispatch 2024-03-26T02:05:08.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82627-24"}]: dispatch 2024-03-26T02:05:08.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi072-82627-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:08.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:08 smithi072 ceph-mon[41619]: pgmap v172: 428 pgs: 40 unknown, 388 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T02:05:08.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:08 smithi072 ceph-mon[41619]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:08.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi072-82404-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:08.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:08 smithi072 ceph-mon[41619]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T02:05:08.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1233630792' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi072-82387-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:08.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:08.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82627-24"}]: dispatch 2024-03-26T02:05:08.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82627-24"}]: dispatch 2024-03-26T02:05:08.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi072-82627-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:09.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:08 smithi173 ceph-mon[33839]: pgmap v172: 428 pgs: 40 unknown, 388 active+clean; 217 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T02:05:09.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:08 smithi173 ceph-mon[33839]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:09.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi072-82404-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:09.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:08 smithi173 ceph-mon[33839]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T02:05:09.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1233630792' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi072-82387-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:09.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:09.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82627-24"}]: dispatch 2024-03-26T02:05:09.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82627-24"}]: dispatch 2024-03-26T02:05:09.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi072-82627-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:09.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1233630792' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi072-82387-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:09.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi072-82627-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:09.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:09 smithi072 ceph-mon[30177]: osdmap e143: 8 total, 8 up, 8 in 2024-03-26T02:05:09.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi072-82627-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi072-82627-24"}]: dispatch 2024-03-26T02:05:09.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:09.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1233630792' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi072-82387-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:09.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi072-82627-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:09.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:09 smithi072 ceph-mon[41619]: osdmap e143: 8 total, 8 up, 8 in 2024-03-26T02:05:09.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi072-82627-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi072-82627-24"}]: dispatch 2024-03-26T02:05:09.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:10.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1233630792' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi072-82387-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:10.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi072-82627-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:10.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:09 smithi173 ceph-mon[33839]: osdmap e143: 8 total, 8 up, 8 in 2024-03-26T02:05:10.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi072-82627-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi072-82627-24"}]: dispatch 2024-03-26T02:05:10.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:10.966 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:05:10 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:05:10.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:10 smithi173 ceph-mon[33839]: pgmap v175: 396 pgs: 40 unknown, 356 active+clean; 592 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-26T02:05:10.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:10.967 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:10 smithi173 ceph-mon[33839]: osdmap e144: 8 total, 8 up, 8 in 2024-03-26T02:05:10.967 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1992141771' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi072-82404-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:10.967 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi072-82404-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:10.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:10 smithi072 ceph-mon[30177]: pgmap v175: 396 pgs: 40 unknown, 356 active+clean; 592 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-26T02:05:10.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:10.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:10 smithi072 ceph-mon[30177]: osdmap e144: 8 total, 8 up, 8 in 2024-03-26T02:05:10.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1992141771' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi072-82404-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:10.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi072-82404-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:10.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:10 smithi072 ceph-mon[41619]: pgmap v175: 396 pgs: 40 unknown, 356 active+clean; 592 KiB data, 1.2 GiB used, 714 GiB / 715 GiB avail 2024-03-26T02:05:10.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:10.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:10 smithi072 ceph-mon[41619]: osdmap e144: 8 total, 8 up, 8 in 2024-03-26T02:05:10.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1992141771' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi072-82404-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:10.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi072-82404-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:11.687 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP api_snapshots_pp: Running main() from gmock_main.cc 2024-03-26T02:05:11.688 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-03-26T02:05:11.688 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-03-26T02:05:11.688 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-03-26T02:05:11.688 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: seed 82705 2024-03-26T02:05:11.688 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-03-26T02:05:11.688 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (2329 ms) 2024-03-26T02:05:11.688 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-03-26T02:05:11.688 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (2310 ms) 2024-03-26T02:05:11.688 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-03-26T02:05:11.688 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (4007 ms) 2024-03-26T02:05:11.688 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-03-26T02:05:11.689 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2066 ms) 2024-03-26T02:05:11.689 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-03-26T02:05:11.689 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (4104 ms) 2024-03-26T02:05:11.689 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (14816 ms total) 2024-03-26T02:05:11.689 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: 2024-03-26T02:05:11.689 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-03-26T02:05:11.689 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-03-26T02:05:11.689 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4880 ms) 2024-03-26T02:05:11.689 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-03-26T02:05:11.689 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (5492 ms) 2024-03-26T02:05:11.689 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-03-26T02:05:11.689 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (6885 ms) 2024-03-26T02:05:11.689 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-03-26T02:05:11.689 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4043 ms) 2024-03-26T02:05:11.690 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-03-26T02:05:11.690 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (1816 ms) 2024-03-26T02:05:11.690 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-26T02:05:11.690 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1775-g5e70124b/rpm/el9/BUILD/ceph-19.0.0-1775-g5e70124b/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-03-26T02:05:11.690 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: 2024-03-26T02:05:11.690 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-03-26T02:05:11.690 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-03-26T02:05:11.690 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi072-82705-7 2024-03-26T02:05:11.690 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: waiting for snaps to purge 2024-03-26T02:05:11.690 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17958 ms) 2024-03-26T02:05:11.690 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (41074 ms total) 2024-03-26T02:05:11.690 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: 2024-03-26T02:05:11.690 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-03-26T02:05:11.690 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-03-26T02:05:11.691 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (9 ms) 2024-03-26T02:05:11.691 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-03-26T02:05:11.691 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (4983 ms) 2024-03-26T02:05:11.691 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (4993 ms total) 2024-03-26T02:05:11.691 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: 2024-03-26T02:05:11.691 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-03-26T02:05:11.691 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-03-26T02:05:11.691 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3260 ms) 2024-03-26T02:05:11.691 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-03-26T02:05:11.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:11 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:05:11.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:11 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi072-82627-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi072-82627-24"}]': finished 2024-03-26T02:05:11.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:11 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi072-82404-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:11.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:11 smithi072 ceph-mon[30177]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T02:05:11.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:11 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:11.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:11 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/460924962' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi072-82387-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:11.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:11 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:05:11.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:11 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi072-82627-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi072-82627-24"}]': finished 2024-03-26T02:05:11.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:11 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi072-82404-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:11.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:11 smithi072 ceph-mon[41619]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T02:05:11.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:11 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:11.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:11 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/460924962' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi072-82387-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:12.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:11 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:05:12.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:11 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi072-82627-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi072-82627-24"}]': finished 2024-03-26T02:05:12.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:11 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi072-82404-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:12.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:11 smithi173 ceph-mon[33839]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T02:05:12.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:11 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:12.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:11 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/460924962' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi072-82387-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:12.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:12 smithi072 ceph-mon[30177]: pgmap v178: 436 pgs: 40 unknown, 32 creating+peering, 364 active+clean; 592 KiB data, 571 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:05:12.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/460924962' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi072-82387-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:12.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:12.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:12 smithi072 ceph-mon[30177]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T02:05:12.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82627-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:12.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:12 smithi072 ceph-mon[41619]: pgmap v178: 436 pgs: 40 unknown, 32 creating+peering, 364 active+clean; 592 KiB data, 571 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:05:12.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/460924962' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi072-82387-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:12.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:12.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:12 smithi072 ceph-mon[41619]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T02:05:12.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82627-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:13.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:12 smithi173 ceph-mon[33839]: pgmap v178: 436 pgs: 40 unknown, 32 creating+peering, 364 active+clean; 592 KiB data, 571 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:05:13.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/460924962' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi072-82387-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:13.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:13.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:12 smithi173 ceph-mon[33839]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T02:05:13.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82627-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:13.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:13 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82627-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:13.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:13 smithi072 ceph-mon[30177]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T02:05:13.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:13 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:13.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:13 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1165695619' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi072-82404-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:13.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:13 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82627-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:13.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:13 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi072-82404-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:13.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:13 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82627-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:13.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:13 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi072-82404-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:13.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:13 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:13.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:13 smithi072 ceph-mon[30177]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T02:05:13.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:13 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2576781945' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi072-82387-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:13.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:13 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82627-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:13.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:13 smithi072 ceph-mon[41619]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T02:05:13.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:13 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:13.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:13 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1165695619' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi072-82404-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:13.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:13 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82627-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:13.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:13 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi072-82404-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:13.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:13 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82627-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:13.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:13 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi072-82404-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:13.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:13 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:13.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:13 smithi072 ceph-mon[41619]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T02:05:13.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:13 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2576781945' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi072-82387-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:14.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:13 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82627-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:14.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:13 smithi173 ceph-mon[33839]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T02:05:14.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:13 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:14.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:13 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1165695619' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi072-82404-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:14.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:13 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82627-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:14.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:13 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi072-82404-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:14.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:13 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82627-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:14.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:13 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi072-82404-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:14.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:13 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:14.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:13 smithi173 ceph-mon[33839]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T02:05:14.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:13 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2576781945' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi072-82387-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:14.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:14 smithi072 ceph-mon[30177]: pgmap v181: 436 pgs: 72 unknown, 364 active+clean; 592 KiB data, 571 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T02:05:14.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:14 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi072-82387-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:14.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:14 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi072-82387-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:14.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:14 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:14.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:14 smithi072 ceph-mon[30177]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T02:05:14.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:14 smithi072 ceph-mon[41619]: pgmap v181: 436 pgs: 72 unknown, 364 active+clean; 592 KiB data, 571 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T02:05:14.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:14 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi072-82387-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:14.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:14 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi072-82387-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:14.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:14 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:14.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:14 smithi072 ceph-mon[41619]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T02:05:15.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:14 smithi173 ceph-mon[33839]: pgmap v181: 436 pgs: 72 unknown, 364 active+clean; 592 KiB data, 571 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T02:05:15.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:14 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi072-82387-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:15.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:14 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi072-82387-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:15.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:14 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:15.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:14 smithi173 ceph-mon[33839]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T02:05:15.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:05:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:05:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:05:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:15 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi072-82492-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T02:05:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:15 smithi072 ceph-mon[30177]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:15 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi072-82492-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-26T02:05:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:15 smithi072 ceph-mon[30177]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T02:05:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2464589009' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi072-82404-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82627-24"}]: dispatch 2024-03-26T02:05:16.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi072-82404-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:15 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi072-82492-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T02:05:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:15 smithi072 ceph-mon[41619]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:15 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi072-82492-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-26T02:05:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:15 smithi072 ceph-mon[41619]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T02:05:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2464589009' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi072-82404-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82627-24"}]: dispatch 2024-03-26T02:05:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi072-82404-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:16.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:15 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi072-82492-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T02:05:16.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:15 smithi173 ceph-mon[33839]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:16.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:15 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi072-82492-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-26T02:05:16.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:15 smithi173 ceph-mon[33839]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T02:05:16.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2464589009' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi072-82404-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:16.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82627-24"}]: dispatch 2024-03-26T02:05:16.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi072-82404-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:16.894 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:16 smithi173 ceph-mon[33839]: pgmap v184: 404 pgs: 404 active+clean; 592 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T02:05:16.894 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:16 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:16.894 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:16 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82627-24"}]': finished 2024-03-26T02:05:16.894 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:16 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi072-82404-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:16.894 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:16 smithi173 ceph-mon[33839]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T02:05:16.894 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:16 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82627-24"}]: dispatch 2024-03-26T02:05:16.895 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:16 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1114043816' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi072-82387-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:16.895 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:16 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4134971250' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:16.895 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:16 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi072-82387-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:16.895 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:16 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:16.895 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:16 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:16.895 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:16 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:16.895 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:16 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T02:05:16.895 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:16 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T02:05:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[30177]: pgmap v184: 404 pgs: 404 active+clean; 592 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T02:05:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82627-24"}]': finished 2024-03-26T02:05:17.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi072-82404-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:17.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[30177]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T02:05:17.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82627-24"}]: dispatch 2024-03-26T02:05:17.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1114043816' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi072-82387-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:17.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4134971250' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:17.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi072-82387-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:17.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:17.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:17.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:17.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T02:05:17.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T02:05:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[41619]: pgmap v184: 404 pgs: 404 active+clean; 592 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T02:05:17.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:17.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82627-24"}]': finished 2024-03-26T02:05:17.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi072-82404-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:17.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[41619]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T02:05:17.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82627-24"}]: dispatch 2024-03-26T02:05:17.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1114043816' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi072-82387-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:17.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4134971250' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:17.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi072-82387-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:17.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:17.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:17.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:17.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T02:05:17.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:16 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T02:05:18.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:18 smithi072 ceph-mon[30177]: pgmap v187: 420 pgs: 64 unknown, 356 active+clean; 592 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:05:18.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82627-24"}]': finished 2024-03-26T02:05:18.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi072-82387-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:18.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi072-82705-16"}]': finished 2024-03-26T02:05:18.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pg_num","val":"11"}]': finished 2024-03-26T02:05:18.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:18 smithi072 ceph-mon[30177]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T02:05:18.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4134971250' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:18.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:18.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:18.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:18.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T02:05:18.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T02:05:18.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:18 smithi072 ceph-mon[41619]: pgmap v187: 420 pgs: 64 unknown, 356 active+clean; 592 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:05:18.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82627-24"}]': finished 2024-03-26T02:05:18.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi072-82387-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:18.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi072-82705-16"}]': finished 2024-03-26T02:05:18.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pg_num","val":"11"}]': finished 2024-03-26T02:05:18.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:18 smithi072 ceph-mon[41619]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T02:05:18.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4134971250' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:18.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:18.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:18.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:18.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T02:05:18.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T02:05:19.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:18 smithi173 ceph-mon[33839]: pgmap v187: 420 pgs: 64 unknown, 356 active+clean; 592 KiB data, 591 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:05:19.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1087454459' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82627-24"}]': finished 2024-03-26T02:05:19.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi072-82387-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:19.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi072-82705-16"}]': finished 2024-03-26T02:05:19.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pg_num","val":"11"}]': finished 2024-03-26T02:05:19.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:18 smithi173 ceph-mon[33839]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T02:05:19.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4134971250' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:19.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:19.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi072-82705-16"}]: dispatch 2024-03-26T02:05:19.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:19.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T02:05:19.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T02:05:19.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi072-82705-16"}]': finished 2024-03-26T02:05:19.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T02:05:19.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[30177]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T02:05:19.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2305485440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi072-82627-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:19.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1630813574' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi072-82404-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:19.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:19.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3133116723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:19.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:19.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3133116723' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:19.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:19.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3133116723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:19.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:19.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2305485440' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi072-82627-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:19.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1630813574' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi072-82404-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:19.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:19.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[30177]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T02:05:19.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi072-82705-16"}]': finished 2024-03-26T02:05:19.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T02:05:19.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[41619]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T02:05:19.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2305485440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi072-82627-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:19.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1630813574' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi072-82404-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:19.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:19.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3133116723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:19.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:19.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3133116723' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:19.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:19.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3133116723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:19.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:19.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2305485440' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi072-82627-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:19.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1630813574' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi072-82404-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:19.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:19.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:19 smithi072 ceph-mon[41619]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T02:05:20.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:19 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi072-82705-16"}]': finished 2024-03-26T02:05:20.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:19 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi072-82492-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T02:05:20.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:19 smithi173 ceph-mon[33839]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T02:05:20.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2305485440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi072-82627-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:20.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1630813574' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi072-82404-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:20.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:20.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3133116723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:20.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:19 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:20.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3133116723' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:20.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:19 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:20.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3133116723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:20.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:19 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:20.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2305485440' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi072-82627-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:20.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1630813574' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi072-82404-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:20.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:19 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:20.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:19 smithi173 ceph-mon[33839]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T02:05:20.879 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:05:20 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:05:21.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:20 smithi072 ceph-mon[41619]: pgmap v190: 420 pgs: 64 unknown, 356 active+clean; 592 KiB data, 591 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:05:21.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3133116723' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:21.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4238817347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi072-82387-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:21.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:20 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi072-82387-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:21.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:21.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:20 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:21.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:20 smithi072 ceph-mon[41619]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:21.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:20 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi072-82387-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:21.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:20 smithi072 ceph-mon[30177]: pgmap v190: 420 pgs: 64 unknown, 356 active+clean; 592 KiB data, 591 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:05:21.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3133116723' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:21.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4238817347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi072-82387-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:21.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:20 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi072-82387-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:21.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:21.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:20 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:21.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:20 smithi072 ceph-mon[30177]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:21.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:20 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi072-82387-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:21.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:20 smithi173 ceph-mon[33839]: pgmap v190: 420 pgs: 64 unknown, 356 active+clean; 592 KiB data, 591 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:05:21.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3133116723' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:21.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4238817347' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi072-82387-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:21.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:20 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi072-82387-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:21.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:21.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:20 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:21.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:20 smithi173 ceph-mon[33839]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:21.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:20 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi072-82387-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:22.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:21 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:05:22.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:22.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:21 smithi072 ceph-mon[30177]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T02:05:22.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:21 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:05:22.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:21 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi072-82492-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T02:05:22.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:21 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]': finished 2024-03-26T02:05:22.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:21 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi072-82492-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-26T02:05:22.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:21 smithi072 ceph-mon[30177]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T02:05:22.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:22.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/226200929' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi072-82404-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:22.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3960104802' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi072-82627-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:22.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:21 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi072-82404-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:22.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:21 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:05:22.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:22.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:21 smithi072 ceph-mon[41619]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T02:05:22.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:21 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:05:22.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:21 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi072-82492-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T02:05:22.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:21 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]': finished 2024-03-26T02:05:22.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:21 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi072-82492-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-26T02:05:22.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:21 smithi072 ceph-mon[41619]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T02:05:22.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:22.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/226200929' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi072-82404-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:22.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3960104802' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi072-82627-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:22.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:21 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi072-82404-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:21 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:05:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:21 smithi173 ceph-mon[33839]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T02:05:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:21 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:05:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:21 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi072-82492-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T02:05:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:21 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]': finished 2024-03-26T02:05:22.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:21 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi072-82492-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-26T02:05:22.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:21 smithi173 ceph-mon[33839]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T02:05:22.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:22.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/226200929' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi072-82404-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:22.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3960104802' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi072-82627-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:22.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:21 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi072-82404-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:23.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:22 smithi072 ceph-mon[41619]: pgmap v193: 388 pgs: 32 creating+peering, 356 active+clean; 592 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T02:05:23.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:22 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3960104802' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi072-82627-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:23.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:22 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi072-82404-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:23.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:22 smithi072 ceph-mon[41619]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T02:05:23.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:22 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:23.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:22 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3308931096' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi072-82387-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:23.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:22 smithi072 ceph-mon[30177]: pgmap v193: 388 pgs: 32 creating+peering, 356 active+clean; 592 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T02:05:23.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:22 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3960104802' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi072-82627-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:23.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:22 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi072-82404-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:23.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:22 smithi072 ceph-mon[30177]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T02:05:23.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:22 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:23.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:22 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3308931096' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi072-82387-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:22 smithi173 ceph-mon[33839]: pgmap v193: 388 pgs: 32 creating+peering, 356 active+clean; 592 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T02:05:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:22 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3960104802' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi072-82627-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:22 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi072-82404-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:22 smithi173 ceph-mon[33839]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T02:05:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:22 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:23.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:22 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3308931096' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi072-82387-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:24.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:24 smithi072 ceph-mon[30177]: pgmap v196: 460 pgs: 104 unknown, 356 active+clean; 592 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T02:05:24.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:24 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3308931096' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi072-82387-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:24.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:24 smithi072 ceph-mon[30177]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T02:05:24.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:24 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:24.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:24 smithi072 ceph-mon[41619]: pgmap v196: 460 pgs: 104 unknown, 356 active+clean; 592 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T02:05:24.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:24 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3308931096' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi072-82387-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:24.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:24 smithi072 ceph-mon[41619]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T02:05:24.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:24 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:25.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:24 smithi173 ceph-mon[33839]: pgmap v196: 460 pgs: 104 unknown, 356 active+clean; 592 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T02:05:25.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:24 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3308931096' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi072-82387-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:25.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:24 smithi173 ceph-mon[33839]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T02:05:25.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:24 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:25.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:05:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:05:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:05:25.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:25 smithi072 ceph-mon[30177]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T02:05:25.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:25.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/565231196' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi072-82404-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:25.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1945895533' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi072-82627-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:25.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:25 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi072-82404-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:25.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:25 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi072-82627-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:25.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/176353717' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:25.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:25 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:25.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/176353717' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:25.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:25 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:25.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/176353717' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi072-82387-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:25.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:25 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi072-82387-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:25.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:25 smithi072 ceph-mon[41619]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T02:05:25.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:25.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/565231196' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi072-82404-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:25.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1945895533' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi072-82627-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:25.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:25 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi072-82404-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:25.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:25 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi072-82627-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:25.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/176353717' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:25.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:25 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:25.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/176353717' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:25.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:25 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:25.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/176353717' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi072-82387-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:25.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:25 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi072-82387-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:26.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:25 smithi173 ceph-mon[33839]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T02:05:26.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:26.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/565231196' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi072-82404-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:26.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1945895533' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi072-82627-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:26.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:25 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi072-82404-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:26.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:25 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi072-82627-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:26.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/176353717' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:26.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:25 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:26.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/176353717' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:26.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:25 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:26.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/176353717' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi072-82387-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:26.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:25 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi072-82387-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:26.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:26 smithi072 ceph-mon[30177]: pgmap v199: 427 pgs: 64 creating+peering, 363 active+clean; 608 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-26T02:05:26.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:26 smithi072 ceph-mon[30177]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:26.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:26 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:26.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:26 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi072-82404-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:26.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:26 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi072-82627-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:26.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:26 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi072-82387-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:26.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:26 smithi072 ceph-mon[30177]: osdmap e160: 8 total, 8 up, 8 in 2024-03-26T02:05:26.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:26 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/176353717' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi072-82387-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:26.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:26 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi072-82387-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:26 smithi072 ceph-mon[41619]: pgmap v199: 427 pgs: 64 creating+peering, 363 active+clean; 608 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-26T02:05:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:26 smithi072 ceph-mon[41619]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:26 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:26 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi072-82404-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:26 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi072-82627-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:26 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi072-82387-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:26 smithi072 ceph-mon[41619]: osdmap e160: 8 total, 8 up, 8 in 2024-03-26T02:05:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:26 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/176353717' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi072-82387-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:26 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi072-82387-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:26 smithi173 ceph-mon[33839]: pgmap v199: 427 pgs: 64 creating+peering, 363 active+clean; 608 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-26T02:05:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:26 smithi173 ceph-mon[33839]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:26 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:26 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi072-82404-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:26 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi072-82627-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:26 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi072-82387-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:27.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:26 smithi173 ceph-mon[33839]: osdmap e160: 8 total, 8 up, 8 in 2024-03-26T02:05:27.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:26 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/176353717' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi072-82387-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:27.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:26 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi072-82387-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:27.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:27 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:27.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:27 smithi072 ceph-mon[30177]: osdmap e161: 8 total, 8 up, 8 in 2024-03-26T02:05:27.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:27 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi072-82627-36"}]: dispatch 2024-03-26T02:05:27.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:27 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi072-82627-36"}]: dispatch 2024-03-26T02:05:27.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:27 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi072-82627-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:27.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:27 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi072-82492-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T02:05:27.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:27 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:27.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:27 smithi072 ceph-mon[41619]: osdmap e161: 8 total, 8 up, 8 in 2024-03-26T02:05:27.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:27 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi072-82627-36"}]: dispatch 2024-03-26T02:05:27.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:27 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi072-82627-36"}]: dispatch 2024-03-26T02:05:27.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:27 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi072-82627-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:27.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:27 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi072-82492-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T02:05:28.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:27 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:28.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:27 smithi173 ceph-mon[33839]: osdmap e161: 8 total, 8 up, 8 in 2024-03-26T02:05:28.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:27 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi072-82627-36"}]: dispatch 2024-03-26T02:05:28.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:27 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi072-82627-36"}]: dispatch 2024-03-26T02:05:28.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:27 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi072-82627-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:28.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:27 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi072-82492-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T02:05:28.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:28 smithi072 ceph-mon[30177]: pgmap v202: 363 pgs: 363 active+clean; 608 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-26T02:05:28.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:28 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:28.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:28 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi072-82387-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi072-82387-27"}]': finished 2024-03-26T02:05:28.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:28 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi072-82627-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:28.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:28 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi072-82492-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-03-26T02:05:28.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:28 smithi072 ceph-mon[30177]: osdmap e162: 8 total, 8 up, 8 in 2024-03-26T02:05:28.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:28 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi072-82627-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi072-82627-36"}]: dispatch 2024-03-26T02:05:28.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:28 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/965633726' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi072-82404-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:28.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:28 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi072-82404-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:28.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:28 smithi072 ceph-mon[41619]: pgmap v202: 363 pgs: 363 active+clean; 608 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-26T02:05:28.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:28 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:28.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:28 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi072-82387-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi072-82387-27"}]': finished 2024-03-26T02:05:28.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:28 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi072-82627-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:28.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:28 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi072-82492-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-03-26T02:05:28.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:28 smithi072 ceph-mon[41619]: osdmap e162: 8 total, 8 up, 8 in 2024-03-26T02:05:28.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:28 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi072-82627-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi072-82627-36"}]: dispatch 2024-03-26T02:05:28.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:28 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/965633726' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi072-82404-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:28.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:28 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi072-82404-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:29.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:28 smithi173 ceph-mon[33839]: pgmap v202: 363 pgs: 363 active+clean; 608 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-26T02:05:29.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:28 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:29.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:28 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi072-82387-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi072-82387-27"}]': finished 2024-03-26T02:05:29.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:28 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi072-82627-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:29.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:28 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi072-82492-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-03-26T02:05:29.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:28 smithi173 ceph-mon[33839]: osdmap e162: 8 total, 8 up, 8 in 2024-03-26T02:05:29.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:28 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi072-82627-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi072-82627-36"}]: dispatch 2024-03-26T02:05:29.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:28 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/965633726' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi072-82404-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:29.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:28 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi072-82404-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:29.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:29 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:29.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:29 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi072-82404-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:29.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:29 smithi072 ceph-mon[30177]: osdmap e163: 8 total, 8 up, 8 in 2024-03-26T02:05:29.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:29 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:29.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:29 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi072-82404-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:29.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:29 smithi072 ceph-mon[41619]: osdmap e163: 8 total, 8 up, 8 in 2024-03-26T02:05:30.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:29 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2246905410' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T02:05:30.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:29 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi072-82404-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:30.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:29 smithi173 ceph-mon[33839]: osdmap e163: 8 total, 8 up, 8 in 2024-03-26T02:05:30.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[30177]: pgmap v205: 403 pgs: 40 unknown, 363 active+clean; 608 KiB data, 597 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:05:30.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi072-82627-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi072-82627-36"}]': finished 2024-03-26T02:05:30.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[30177]: osdmap e164: 8 total, 8 up, 8 in 2024-03-26T02:05:30.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/176353717' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:30.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:30.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1711694665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:30.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:30.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1711694665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:30.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:05:30.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:30.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:05:30.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1711694665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi072-82492-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:30.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi072-82492-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:30.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi072-82387-27"}]': finished 2024-03-26T02:05:30.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:05:30.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi072-82492-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:30.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[30177]: osdmap e165: 8 total, 8 up, 8 in 2024-03-26T02:05:30.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-13"}]: dispatch 2024-03-26T02:05:30.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2521679588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi072-82404-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:30.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[41619]: pgmap v205: 403 pgs: 40 unknown, 363 active+clean; 608 KiB data, 597 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:05:30.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi072-82627-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi072-82627-36"}]': finished 2024-03-26T02:05:30.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[41619]: osdmap e164: 8 total, 8 up, 8 in 2024-03-26T02:05:30.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/176353717' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:30.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:30.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1711694665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:30.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:30.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1711694665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:30.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:05:30.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:30.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:05:30.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1711694665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi072-82492-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:30.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi072-82492-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:30.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi072-82387-27"}]': finished 2024-03-26T02:05:30.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:05:30.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi072-82492-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:30.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[41619]: osdmap e165: 8 total, 8 up, 8 in 2024-03-26T02:05:30.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-13"}]: dispatch 2024-03-26T02:05:30.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2521679588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi072-82404-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:30.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi072-82404-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:30.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/176353717' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:30.973 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1711694665' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi072-82492-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:30.973 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-13"}]: dispatch 2024-03-26T02:05:30.973 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:30.973 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi072-82492-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:30.973 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi072-82404-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:30.973 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/176353717' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:30.973 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1711694665' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi072-82492-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:30.973 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-13"}]: dispatch 2024-03-26T02:05:30.973 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:30.973 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi072-82492-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:31.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:05:30 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:05:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:30 smithi173 ceph-mon[33839]: pgmap v205: 403 pgs: 40 unknown, 363 active+clean; 608 KiB data, 597 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:05:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi072-82627-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi072-82627-36"}]': finished 2024-03-26T02:05:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:30 smithi173 ceph-mon[33839]: osdmap e164: 8 total, 8 up, 8 in 2024-03-26T02:05:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/176353717' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1711694665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1711694665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:05:31.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:31.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:05:31.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1711694665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi072-82492-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:31.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi072-82492-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:31.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi072-82387-27"}]': finished 2024-03-26T02:05:31.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:05:31.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi072-82492-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:31.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:30 smithi173 ceph-mon[33839]: osdmap e165: 8 total, 8 up, 8 in 2024-03-26T02:05:31.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-13"}]: dispatch 2024-03-26T02:05:31.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2521679588' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi072-82404-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:31.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi072-82404-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:31.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/176353717' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:31.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1711694665' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi072-82492-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:31.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-13"}]: dispatch 2024-03-26T02:05:31.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi072-82387-27"}]: dispatch 2024-03-26T02:05:31.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi072-82492-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:31.626 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ OK ] LibRadosSnaps api_aio: Running main() from gmock_main.cc 2024-03-26T02:05:31.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-03-26T02:05:31.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [----------] Global test environment set-up. 2024-03-26T02:05:31.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-03-26T02:05:31.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-03-26T02:05:31.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAio.TooBig (10392 ms) 2024-03-26T02:05:31.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-03-26T02:05:31.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (6309 ms) 2024-03-26T02:05:31.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-03-26T02:05:31.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (4062 ms) 2024-03-26T02:05:31.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-03-26T02:05:31.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (3843 ms) 2024-03-26T02:05:31.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-03-26T02:05:31.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (5452 ms) 2024-03-26T02:05:31.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-03-26T02:05:31.627 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (3162 ms) 2024-03-26T02:05:31.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-03-26T02:05:31.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (4467 ms) 2024-03-26T02:05:31.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-03-26T02:05:31.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3418 ms) 2024-03-26T02:05:31.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-03-26T02:05:31.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3660 ms) 2024-03-26T02:05:31.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-03-26T02:05:31.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3025 ms) 2024-03-26T02:05:31.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-03-26T02:05:31.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3000 ms) 2024-03-26T02:05:31.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-03-26T02:05:31.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (2786 ms) 2024-03-26T02:05:31.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-03-26T02:05:31.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (2992 ms) 2024-03-26T02:05:31.628 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-03-26T02:05:31.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3015 ms) 2024-03-26T02:05:31.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-03-26T02:05:31.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAio.Flush (2985 ms) 2024-03-26T02:05:31.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-03-26T02:05:31.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3007 ms) 2024-03-26T02:05:31.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-03-26T02:05:31.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3047 ms) 2024-03-26T02:05:31.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-03-26T02:05:31.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (2959 ms) 2024-03-26T02:05:31.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-03-26T02:05:31.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (2997 ms) 2024-03-26T02:05:31.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-03-26T02:05:31.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3026 ms) 2024-03-26T02:05:31.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-03-26T02:05:31.629 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3266 ms) 2024-03-26T02:05:31.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-03-26T02:05:31.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (2783 ms) 2024-03-26T02:05:31.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-03-26T02:05:31.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (2919 ms) 2024-03-26T02:05:31.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-03-26T02:05:31.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3007 ms) 2024-03-26T02:05:31.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-03-26T02:05:31.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3020 ms) 2024-03-26T02:05:31.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-03-26T02:05:31.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3002 ms) 2024-03-26T02:05:31.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [----------] 26 tests from LibRadosAio (95602 ms total) 2024-03-26T02:05:31.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: 2024-03-26T02:05:31.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-03-26T02:05:31.630 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-03-26T02:05:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:31 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:05:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:31 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:05:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:31 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi072-82404-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:31 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-13"}]': finished 2024-03-26T02:05:32.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:31 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:05:32.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:31 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:05:32.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:31 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi072-82404-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:32.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:31 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-13"}]': finished 2024-03-26T02:05:32.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:31 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:05:32.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:31 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:05:32.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:31 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi072-82404-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:32.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:31 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-13"}]': finished 2024-03-26T02:05:33.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[41619]: pgmap v208: 372 pgs: 32 unknown, 8 creating+peering, 332 active+clean; 607 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 2 op/s 2024-03-26T02:05:33.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi072-82387-27"}]': finished 2024-03-26T02:05:33.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2556928150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:33.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[41619]: osdmap e166: 8 total, 8 up, 8 in 2024-03-26T02:05:33.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[41619]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:33.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi072-82627-36"}]: dispatch 2024-03-26T02:05:33.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:33.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2556928150' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:33.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:33.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2556928150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi072-82387-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:33.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi072-82387-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:33.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi072-82492-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi072-82492-2"}]': finished 2024-03-26T02:05:33.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi072-82627-36"}]': finished 2024-03-26T02:05:33.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi072-82387-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:33.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[41619]: osdmap e167: 8 total, 8 up, 8 in 2024-03-26T02:05:33.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi072-82627-36"}]: dispatch 2024-03-26T02:05:33.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2556928150' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi072-82387-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:33.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi072-82387-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:33.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[30177]: pgmap v208: 372 pgs: 32 unknown, 8 creating+peering, 332 active+clean; 607 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 2 op/s 2024-03-26T02:05:33.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi072-82387-27"}]': finished 2024-03-26T02:05:33.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2556928150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:33.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[30177]: osdmap e166: 8 total, 8 up, 8 in 2024-03-26T02:05:33.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[30177]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:33.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi072-82627-36"}]: dispatch 2024-03-26T02:05:33.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:33.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2556928150' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:33.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:33.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2556928150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi072-82387-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:33.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi072-82387-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:33.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi072-82492-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi072-82492-2"}]': finished 2024-03-26T02:05:33.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi072-82627-36"}]': finished 2024-03-26T02:05:33.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi072-82387-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:33.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[30177]: osdmap e167: 8 total, 8 up, 8 in 2024-03-26T02:05:33.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi072-82627-36"}]: dispatch 2024-03-26T02:05:33.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2556928150' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi072-82387-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:33.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:32 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi072-82387-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:33.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:32 smithi173 ceph-mon[33839]: pgmap v208: 372 pgs: 32 unknown, 8 creating+peering, 332 active+clean; 607 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 2 op/s 2024-03-26T02:05:33.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:32 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi072-82387-27"}]': finished 2024-03-26T02:05:33.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2556928150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:33.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:32 smithi173 ceph-mon[33839]: osdmap e166: 8 total, 8 up, 8 in 2024-03-26T02:05:33.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:32 smithi173 ceph-mon[33839]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:33.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi072-82627-36"}]: dispatch 2024-03-26T02:05:33.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:32 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:33.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2556928150' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:33.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:32 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:33.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2556928150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi072-82387-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:33.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:32 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi072-82387-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:33.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:32 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi072-82492-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi072-82492-2"}]': finished 2024-03-26T02:05:33.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi072-82627-36"}]': finished 2024-03-26T02:05:33.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:32 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi072-82387-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:33.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:32 smithi173 ceph-mon[33839]: osdmap e167: 8 total, 8 up, 8 in 2024-03-26T02:05:33.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi072-82627-36"}]: dispatch 2024-03-26T02:05:33.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2556928150' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi072-82387-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:33.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:32 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi072-82387-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:33.684 INFO:tasks.workunit.client.0.smithi072.stdout:xpected=11 2024-03-26T02:05:33.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:863748b0:::15:head 2024-03-26T02:05:33.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : cursor()=13:863748b0:::15:head expected=13:863748b0:::15:head 2024-03-26T02:05:33.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:863748b0:::15:head -> 15 2024-03-26T02:05:33.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : entry=15 expected=15 2024-03-26T02:05:33.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:6cac518f:::0:head 2024-03-26T02:05:33.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : cursor()=13:6cac518f:::0:head expected=13:6cac518f:::0:head 2024-03-26T02:05:33.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:6cac518f:::0:head -> 0 2024-03-26T02:05:33.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : entry=0 expected=0 2024-03-26T02:05:33.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:52ea6a34:::10:head 2024-03-26T02:05:33.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : cursor()=13:52ea6a34:::10:head expected=13:52ea6a34:::10:head 2024-03-26T02:05:33.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:52ea6a34:::10:head -> 10 2024-03-26T02:05:33.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : entry=10 expected=10 2024-03-26T02:05:33.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:f905c69b:::2:head 2024-03-26T02:05:33.685 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : cursor()=13:f905c69b:::2:head expected=13:f905c69b:::2:head 2024-03-26T02:05:33.686 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:f905c69b:::2:head -> 2 2024-03-26T02:05:33.686 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : entry=2 expected=2 2024-03-26T02:05:33.686 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:e960b815:::9:head 2024-03-26T02:05:33.686 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : cursor()=13:e960b815:::9:head expected=13:e960b815:::9:head 2024-03-26T02:05:33.686 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:e960b815:::9:head -> 9 2024-03-26T02:05:33.686 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : entry=9 expected=9 2024-03-26T02:05:33.686 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:566253c9:::13:head 2024-03-26T02:05:33.686 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : cursor()=13:566253c9:::13:head expected=13:566253c9:::13:head 2024-03-26T02:05:33.686 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:566253c9:::13:head -> 13 2024-03-26T02:05:33.686 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : entry=13 expected=13 2024-03-26T02:05:33.686 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:62a1935d:::14:head 2024-03-26T02:05:33.686 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : cursor()=13:62a1935d:::14:head expected=13:62a1935d:::14:head 2024-03-26T02:05:33.686 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:62a1935d:::14:head -> 14 2024-03-26T02:05:33.687 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : entry=14 expected=14 2024-03-26T02:05:33.687 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : seek to 13:5c6b0b28:::7:head 2024-03-26T02:05:33.687 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : cursor()=13:5c6b0b28:::7:head expected=13:5c6b0b28:::7:head 2024-03-26T02:05:33.687 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: > 13:5c6b0b28:::7:head -> 7 2024-03-26T02:05:33.687 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: : entry=7 expected=7 2024-03-26T02:05:33.687 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (173 ms) 2024-03-26T02:05:33.687 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-03-26T02:05:33.687 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (78473 ms) 2024-03-26T02:05:33.687 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-03-26T02:05:33.687 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: split 0/5 -> MIN 13:33333333::::head 2024-03-26T02:05:33.687 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: split 1/5 -> 13:33333333::::head 13:66666666::::head 2024-03-26T02:05:33.687 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: split 2/5 -> 13:66666666::::head 13:99999999::::head 2024-03-26T02:05:33.687 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: split 3/5 -> 13:99999999::::head 13:cccccccc::::head 2024-03-26T02:05:33.687 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: split 4/5 -> 13:cccccccc::::head MAX 2024-03-26T02:05:33.687 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (12011 ms) 2024-03-26T02:05:33.687 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [----------] 7 tests from LibRadosList (90917 ms total) 2024-03-26T02:05:33.688 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 2024-03-26T02:05:33.688 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-03-26T02:05:33.688 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-03-26T02:05:33.688 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1030 ms) 2024-03-26T02:05:33.688 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-03-26T02:05:33.688 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: myset foo1,foo2,foo3 2024-03-26T02:05:33.688 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: foo1 2024-03-26T02:05:33.688 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: foo2 2024-03-26T02:05:33.688 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: foo3 2024-03-26T02:05:33.688 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: myset foo1,foo4,foo5 2024-03-26T02:05:33.688 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: foo4 2024-03-26T02:05:33.688 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: foo5 2024-03-26T02:05:33.688 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: foo1 2024-03-26T02:05:33.688 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: myset foo6,foo7 2024-03-26T02:05:33.689 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: foo7 2024-03-26T02:05:33.689 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: foo6 2024-03-26T02:05:33.689 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-26T02:05:33.689 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: ns1:foo4 2024-03-26T02:05:33.689 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: ns1:foo5 2024-03-26T02:05:33.689 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: ns2:foo7 2024-03-26T02:05:33.689 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: ns2:foo6 2024-03-26T02:05:33.689 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: ns1:foo1 2024-03-26T02:05:33.689 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: :foo1 2024-03-26T02:05:33.689 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: :foo2 2024-03-26T02:05:33.689 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: :foo3 2024-03-26T02:05:33.689 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (32 ms) 2024-03-26T02:05:33.884 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [ RUN ] LibRadosListEC.Lis(67820 ms total) 2024-03-26T02:05:33.884 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: 2024-03-26T02:05:33.884 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-03-26T02:05:33.884 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-03-26T02:05:33.884 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (112 ms) 2024-03-26T02:05:33.884 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (112 ms total) 2024-03-26T02:05:33.884 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: 2024-03-26T02:05:33.884 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-03-26T02:05:33.884 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-03-26T02:05:33.884 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (39 ms) 2024-03-26T02:05:33.884 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-03-26T02:05:33.885 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (3 ms) 2024-03-26T02:05:33.885 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (42 ms total) 2024-03-26T02:05:33.885 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: 2024-03-26T02:05:33.885 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-03-26T02:05:33.885 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-03-26T02:05:33.885 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (36 ms) 2024-03-26T02:05:33.885 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-03-26T02:05:33.885 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (6 ms) 2024-03-26T02:05:33.885 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (42 ms total) 2024-03-26T02:05:33.885 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: 2024-03-26T02:05:33.885 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-03-26T02:05:33.885 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-03-26T02:05:33.885 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (36 ms) 2024-03-26T02:05:33.885 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-03-26T02:05:33.886 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (5 ms) 2024-03-26T02:05:33.886 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (41 ms total) 2024-03-26T02:05:33.886 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: 2024-03-26T02:05:33.886 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-03-26T02:05:33.886 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-03-26T02:05:33.886 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (964 ms) 2024-03-26T02:05:33.886 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (964 ms total) 2024-03-26T02:05:33.886 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: 2024-03-26T02:05:33.886 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-03-26T02:05:33.886 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (104747 ms total) 2024-03-26T02:05:33.886 INFO:tasks.workunit.client.0.smithi072.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-03-26T02:05:34.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:34 smithi072 ceph-mon[30177]: pgmap v211: 308 pgs: 8 unknown, 300 active+clean; 607 KiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:05:34.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:34 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi072-82627-36"}]': finished 2024-03-26T02:05:34.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:34 smithi072 ceph-mon[30177]: osdmap e168: 8 total, 8 up, 8 in 2024-03-26T02:05:34.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:34 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3124328836' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi072-82404-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:34.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:34 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:34.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:34 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:34.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:34 smithi072 ceph-mon[41619]: pgmap v211: 308 pgs: 8 unknown, 300 active+clean; 607 KiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:05:34.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:34 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi072-82627-36"}]': finished 2024-03-26T02:05:34.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:34 smithi072 ceph-mon[41619]: osdmap e168: 8 total, 8 up, 8 in 2024-03-26T02:05:34.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:34 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3124328836' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi072-82404-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:34.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:34 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:34.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:34 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:35.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:34 smithi173 ceph-mon[33839]: pgmap v211: 308 pgs: 8 unknown, 300 active+clean; 607 KiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:05:35.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:34 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/316168243' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi072-82627-36"}]': finished 2024-03-26T02:05:35.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:34 smithi173 ceph-mon[33839]: osdmap e168: 8 total, 8 up, 8 in 2024-03-26T02:05:35.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:34 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3124328836' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi072-82404-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:35.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:34 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:35.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:34 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:35.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:05:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:05:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:05:35.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:35 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi072-82387-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi072-82387-28"}]': finished 2024-03-26T02:05:35.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:35 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3124328836' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi072-82404-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:35.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:35 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:35.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:35 smithi072 ceph-mon[30177]: osdmap e169: 8 total, 8 up, 8 in 2024-03-26T02:05:35.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:35 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1711694665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:35.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:35 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:35.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:35 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3133116723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:35.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:35 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:35.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:35 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi072-82492-2"}]': finished 2024-03-26T02:05:35.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:35 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]': finished 2024-03-26T02:05:35.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:35 smithi072 ceph-mon[30177]: osdmap e170: 8 total, 8 up, 8 in 2024-03-26T02:05:35.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:35 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1711694665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:35.972 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:35 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:35.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:35 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi072-82387-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi072-82387-28"}]': finished 2024-03-26T02:05:35.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:35 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3124328836' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi072-82404-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:35.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:35 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:35.973 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:35 smithi072 ceph-mon[41619]: osdmap e169: 8 total, 8 up, 8 in 2024-03-26T02:05:35.973 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:35 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1711694665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:35.973 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:35 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:35.973 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:35 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3133116723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:35.973 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:35 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:35.973 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:35 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi072-82492-2"}]': finished 2024-03-26T02:05:35.973 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:35 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]': finished 2024-03-26T02:05:35.973 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:35 smithi072 ceph-mon[41619]: osdmap e170: 8 total, 8 up, 8 in 2024-03-26T02:05:35.973 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:35 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1711694665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:35.973 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:35 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:36.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:35 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi072-82387-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi072-82387-28"}]': finished 2024-03-26T02:05:36.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:35 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3124328836' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi072-82404-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:36.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:35 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:36.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:35 smithi173 ceph-mon[33839]: osdmap e169: 8 total, 8 up, 8 in 2024-03-26T02:05:36.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:35 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1711694665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:36.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:35 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:36.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:35 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3133116723' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:36.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:35 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:36.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:35 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi072-82492-2"}]': finished 2024-03-26T02:05:36.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:35 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]': finished 2024-03-26T02:05:36.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:35 smithi173 ceph-mon[33839]: osdmap e170: 8 total, 8 up, 8 in 2024-03-26T02:05:36.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:35 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1711694665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:36.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:35 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi072-82492-2"}]: dispatch 2024-03-26T02:05:36.633 INFO:tasks.workunit.client.0.smithi072.stdout:hotsECPP.SnapRemovePP (1778 ms) 2024-03-26T02:05:36.633 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-03-26T02:05:36.634 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (1997 ms) 2024-03-26T02:05:36.634 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-03-26T02:05:36.634 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (1928 ms) 2024-03-26T02:05:36.634 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (8963 ms total) 2024-03-26T02:05:36.634 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: 2024-03-26T02:05:36.634 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-03-26T02:05:36.634 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-03-26T02:05:36.634 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4050 ms) 2024-03-26T02:05:36.634 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-03-26T02:05:36.634 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3993 ms) 2024-03-26T02:05:36.635 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-03-26T02:05:36.635 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4201 ms) 2024-03-26T02:05:36.635 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (12244 ms total) 2024-03-26T02:05:36.635 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: 2024-03-26T02:05:36.635 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-03-26T02:05:36.635 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (107484 ms total) 2024-03-26T02:05:36.635 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-03-26T02:05:36.635 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-03-26T02:05:36.635 INFO:tasks.workunit.client.0.smithi072.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-26T02:05:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[30177]: pgmap v214: 364 pgs: 8 creating+peering, 356 active+clean; 583 KiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:05:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3133116723' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:05:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:05:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:05:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:05:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi072-82492-2"}]': finished 2024-03-26T02:05:37.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]': finished 2024-03-26T02:05:37.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:05:37.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[30177]: osdmap e171: 8 total, 8 up, 8 in 2024-03-26T02:05:37.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-15"}]: dispatch 2024-03-26T02:05:37.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2556928150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:37.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-15"}]: dispatch 2024-03-26T02:05:37.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:37.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2627976207' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi072-82404-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:37.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi072-82404-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:37.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[41619]: pgmap v214: 364 pgs: 8 creating+peering, 356 active+clean; 583 KiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:05:37.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3133116723' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:37.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:37.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:05:37.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:05:37.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:05:37.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:05:37.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi072-82492-2"}]': finished 2024-03-26T02:05:37.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]': finished 2024-03-26T02:05:37.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:05:37.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[41619]: osdmap e171: 8 total, 8 up, 8 in 2024-03-26T02:05:37.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-15"}]: dispatch 2024-03-26T02:05:37.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2556928150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:37.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-15"}]: dispatch 2024-03-26T02:05:37.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:37.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2627976207' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi072-82404-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:37.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:36 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi072-82404-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:36 smithi173 ceph-mon[33839]: pgmap v214: 364 pgs: 8 creating+peering, 356 active+clean; 583 KiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:05:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:36 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3133116723' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:36 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]: dispatch 2024-03-26T02:05:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:36 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:05:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:36 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:05:37.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:36 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:05:37.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:36 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:05:37.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:36 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi072-82492-2"}]': finished 2024-03-26T02:05:37.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:36 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi072-82705-21"}]': finished 2024-03-26T02:05:37.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:36 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:05:37.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:36 smithi173 ceph-mon[33839]: osdmap e171: 8 total, 8 up, 8 in 2024-03-26T02:05:37.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:36 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-15"}]: dispatch 2024-03-26T02:05:37.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:36 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2556928150' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:37.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:36 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-15"}]: dispatch 2024-03-26T02:05:37.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:36 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:37.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:36 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2627976207' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi072-82404-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:37.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:36 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi072-82404-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:38.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:38 smithi072 ceph-mon[41619]: pgmap v217: 356 pgs: 32 unknown, 324 active+clean; 583 KiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:05:38.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:38 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-15"}]': finished 2024-03-26T02:05:38.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:38 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi072-82387-28"}]': finished 2024-03-26T02:05:38.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:38 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi072-82404-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:38.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:38 smithi072 ceph-mon[41619]: osdmap e172: 8 total, 8 up, 8 in 2024-03-26T02:05:38.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:38 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4160247721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82492-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:38.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:38 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2556928150' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:38.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:38 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-15", "mode": "writeback"}]: dispatch 2024-03-26T02:05:38.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:38 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:38.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:38 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-15", "mode": "writeback"}]: dispatch 2024-03-26T02:05:38.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:38 smithi072 ceph-mon[30177]: pgmap v217: 356 pgs: 32 unknown, 324 active+clean; 583 KiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:05:38.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:38 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-15"}]': finished 2024-03-26T02:05:38.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:38 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi072-82387-28"}]': finished 2024-03-26T02:05:38.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:38 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi072-82404-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:38.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:38 smithi072 ceph-mon[30177]: osdmap e172: 8 total, 8 up, 8 in 2024-03-26T02:05:38.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:38 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4160247721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82492-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:38.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:38 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2556928150' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:38.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:38 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-15", "mode": "writeback"}]: dispatch 2024-03-26T02:05:38.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:38 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:38.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:38 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-15", "mode": "writeback"}]: dispatch 2024-03-26T02:05:39.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:38 smithi173 ceph-mon[33839]: pgmap v217: 356 pgs: 32 unknown, 324 active+clean; 583 KiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:05:39.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:38 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-15"}]': finished 2024-03-26T02:05:39.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:38 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi072-82387-28"}]': finished 2024-03-26T02:05:39.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:38 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi072-82404-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:39.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:38 smithi173 ceph-mon[33839]: osdmap e172: 8 total, 8 up, 8 in 2024-03-26T02:05:39.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:38 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4160247721' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82492-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:39.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:38 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2556928150' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:39.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:38 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-15", "mode": "writeback"}]: dispatch 2024-03-26T02:05:39.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:38 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi072-82387-28"}]: dispatch 2024-03-26T02:05:39.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:38 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-15", "mode": "writeback"}]: dispatch 2024-03-26T02:05:39.666 INFO:tasks.workunit.client.0.smithi072.stdout:tObjectsStart 2024-03-26T02:05:39.666 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 1 0 2024-03-26T02:05:39.666 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 10 0 2024-03-26T02:05:39.666 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 13 0 2024-03-26T02:05:39.667 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 7 0 2024-03-26T02:05:39.667 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 14 0 2024-03-26T02:05:39.667 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 0 0 2024-03-26T02:05:39.667 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 15 0 2024-03-26T02:05:39.667 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 11 0 2024-03-26T02:05:39.667 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 5 0 2024-03-26T02:05:39.667 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 8 0 2024-03-26T02:05:39.667 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 6 0 2024-03-26T02:05:39.667 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 3 0 2024-03-26T02:05:39.667 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 4 0 2024-03-26T02:05:39.667 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 12 0 2024-03-26T02:05:39.668 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 9 0 2024-03-26T02:05:39.668 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 2 0 2024-03-26T02:05:39.668 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-26T02:05:39.668 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (44 ms) 2024-03-26T02:05:39.668 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [----------] 3 tests from LibRadosListEC (1106 ms total) 2024-03-26T02:05:39.668 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 2024-03-26T02:05:39.668 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-03-26T02:05:39.668 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-03-26T02:05:39.668 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3038 ms) 2024-03-26T02:05:39.668 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [----------] 1 test from LibRadosListNP (3038 ms total) 2024-03-26T02:05:39.668 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: 2024-03-26T02:05:39.668 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [----------] Global test environment tear-down 2024-03-26T02:05:39.668 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [==========] 11 tests from 3 test suites ran. (110562 ms total) 2024-03-26T02:05:39.669 INFO:tasks.workunit.client.0.smithi072.stdout: api_list: [ PASSED ] 11 tests. 2024-03-26T02:05:39.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:05:39.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4160247721' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82492-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:39.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi072-82387-28"}]': finished 2024-03-26T02:05:39.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-15", "mode": "writeback"}]': finished 2024-03-26T02:05:39.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[30177]: osdmap e173: 8 total, 8 up, 8 in 2024-03-26T02:05:39.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3636275255' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:39.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:39.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3636275255' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:39.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:39.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:05:39.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3636275255' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi072-82387-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:39.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi072-82387-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:39.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4160247721' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi072-82492-3","pool2":"test-rados-api-smithi072-82492-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T02:05:39.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:05:39.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:05:39.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:05:39.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:05:39.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4160247721' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82492-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:39.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi072-82387-28"}]': finished 2024-03-26T02:05:39.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-15", "mode": "writeback"}]': finished 2024-03-26T02:05:39.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[41619]: osdmap e173: 8 total, 8 up, 8 in 2024-03-26T02:05:39.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3636275255' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:39.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:39.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3636275255' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:39.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:39.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:05:39.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3636275255' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi072-82387-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:39.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi072-82387-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:39.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4160247721' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi072-82492-3","pool2":"test-rados-api-smithi072-82492-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T02:05:39.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:05:39.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:05:39.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:39 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:05:40.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:39 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:05:40.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4160247721' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82492-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:40.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:39 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi072-82387-28"}]': finished 2024-03-26T02:05:40.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:39 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-15", "mode": "writeback"}]': finished 2024-03-26T02:05:40.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:39 smithi173 ceph-mon[33839]: osdmap e173: 8 total, 8 up, 8 in 2024-03-26T02:05:40.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3636275255' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:40.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:39 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:40.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3636275255' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:40.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:39 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:40.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:39 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:05:40.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3636275255' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi072-82387-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:40.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:39 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi072-82387-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:40.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4160247721' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi072-82492-3","pool2":"test-rados-api-smithi072-82492-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T02:05:40.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:39 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:05:40.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:39 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:05:40.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:39 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:05:40.912 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:05:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:05:41.177 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:40 smithi173 ceph-mon[33839]: pgmap v220: 356 pgs: 32 unknown, 324 active+clean; 583 KiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:05:41.177 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:40 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:41.178 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:40 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi072-82387-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:41.178 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4160247721' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi072-82492-3","pool2":"test-rados-api-smithi072-82492-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-26T02:05:41.178 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:40 smithi173 ceph-mon[33839]: osdmap e174: 8 total, 8 up, 8 in 2024-03-26T02:05:41.178 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3362422711' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi072-82404-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:41.178 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3636275255' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi072-82387-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:41.178 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:40 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi072-82387-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:41.178 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:05:41.178 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:40 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:05:41.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:40 smithi072 ceph-mon[41619]: pgmap v220: 356 pgs: 32 unknown, 324 active+clean; 583 KiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:05:41.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:40 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:41.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:40 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi072-82387-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:41.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4160247721' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi072-82492-3","pool2":"test-rados-api-smithi072-82492-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-26T02:05:41.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:40 smithi072 ceph-mon[41619]: osdmap e174: 8 total, 8 up, 8 in 2024-03-26T02:05:41.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3362422711' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi072-82404-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:41.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3636275255' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi072-82387-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:41.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:40 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi072-82387-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:05:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:40 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:05:41.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:40 smithi072 ceph-mon[30177]: pgmap v220: 356 pgs: 32 unknown, 324 active+clean; 583 KiB data, 603 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:05:41.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:40 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:41.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:40 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi072-82387-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:41.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4160247721' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi072-82492-3","pool2":"test-rados-api-smithi072-82492-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-26T02:05:41.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:40 smithi072 ceph-mon[30177]: osdmap e174: 8 total, 8 up, 8 in 2024-03-26T02:05:41.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3362422711' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi072-82404-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:41.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3636275255' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi072-82387-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:41.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:40 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi072-82387-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:41.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:05:41.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:40 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:05:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:41 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:05:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:41 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3362422711' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi072-82404-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:41 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:05:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:41 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-15"}]: dispatch 2024-03-26T02:05:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:41 smithi072 ceph-mon[30177]: osdmap e175: 8 total, 8 up, 8 in 2024-03-26T02:05:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:41 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-15"}]: dispatch 2024-03-26T02:05:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:41 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:05:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:41 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi072-82387-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi072-82387-29"}]': finished 2024-03-26T02:05:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:41 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-15"}]': finished 2024-03-26T02:05:42.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:41 smithi072 ceph-mon[30177]: osdmap e176: 8 total, 8 up, 8 in 2024-03-26T02:05:42.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:41 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:05:42.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:41 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3362422711' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi072-82404-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:42.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:41 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:05:42.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:41 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-15"}]: dispatch 2024-03-26T02:05:42.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:41 smithi072 ceph-mon[41619]: osdmap e175: 8 total, 8 up, 8 in 2024-03-26T02:05:42.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:41 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-15"}]: dispatch 2024-03-26T02:05:42.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:41 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:05:42.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:41 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi072-82387-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi072-82387-29"}]': finished 2024-03-26T02:05:42.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:41 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-15"}]': finished 2024-03-26T02:05:42.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:41 smithi072 ceph-mon[41619]: osdmap e176: 8 total, 8 up, 8 in 2024-03-26T02:05:42.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:41 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:05:42.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:41 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3362422711' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi072-82404-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:42.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:41 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:05:42.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:41 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-15"}]: dispatch 2024-03-26T02:05:42.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:41 smithi173 ceph-mon[33839]: osdmap e175: 8 total, 8 up, 8 in 2024-03-26T02:05:42.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:41 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-15"}]: dispatch 2024-03-26T02:05:42.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:41 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:05:42.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:41 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi072-82387-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi072-82387-29"}]': finished 2024-03-26T02:05:42.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:41 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-15"}]': finished 2024-03-26T02:05:42.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:41 smithi173 ceph-mon[33839]: osdmap e176: 8 total, 8 up, 8 in 2024-03-26T02:05:43.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:42 smithi072 ceph-mon[41619]: pgmap v223: 356 pgs: 32 creating+peering, 324 active+clean; 583 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:05:43.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:42 smithi072 ceph-mon[41619]: osdmap e177: 8 total, 8 up, 8 in 2024-03-26T02:05:43.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:42 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3451572927' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi072-82404-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:43.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:42 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi072-82404-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:43.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:42 smithi072 ceph-mon[30177]: pgmap v223: 356 pgs: 32 creating+peering, 324 active+clean; 583 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:05:43.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:42 smithi072 ceph-mon[30177]: osdmap e177: 8 total, 8 up, 8 in 2024-03-26T02:05:43.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:42 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3451572927' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi072-82404-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:43.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:42 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi072-82404-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:42 smithi173 ceph-mon[33839]: pgmap v223: 356 pgs: 32 creating+peering, 324 active+clean; 583 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:05:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:42 smithi173 ceph-mon[33839]: osdmap e177: 8 total, 8 up, 8 in 2024-03-26T02:05:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:42 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3451572927' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi072-82404-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:42 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi072-82404-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:44.673 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-03-26T02:05:44.673 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-03-26T02:05:44.673 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-03-26T02:05:44.673 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-03-26T02:05:44.673 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-03-26T02:05:44.673 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (10388 ms) 2024-03-26T02:05:44.674 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-03-26T02:05:44.674 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (24028 ms) 2024-03-26T02:05:44.674 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-03-26T02:05:44.674 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (8125 ms) 2024-03-26T02:05:44.674 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-03-26T02:05:44.674 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (3219 ms) 2024-03-26T02:05:44.674 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-03-26T02:05:44.674 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (4030 ms) 2024-03-26T02:05:44.674 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-03-26T02:05:44.674 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (2773 ms) 2024-03-26T02:05:44.674 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-03-26T02:05:44.674 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (3022 ms) 2024-03-26T02:05:44.674 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-03-26T02:05:44.675 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (2987 ms) 2024-03-26T02:05:44.675 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-03-26T02:05:44.675 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3244 ms) 2024-03-26T02:05:44.675 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-03-26T02:05:44.675 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (2758 ms) 2024-03-26T02:05:44.675 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-03-26T02:05:44.675 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3007 ms) 2024-03-26T02:05:44.675 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-03-26T02:05:44.675 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3052 ms) 2024-03-26T02:05:44.675 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-03-26T02:05:44.675 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (2940 ms) 2024-03-26T02:05:44.675 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-03-26T02:05:44.675 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3265 ms) 2024-03-26T02:05:44.675 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-03-26T02:05:44.676 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (2776 ms) 2024-03-26T02:05:44.676 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-03-26T02:05:44.676 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3022 ms) 2024-03-26T02:05:44.676 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-03-26T02:05:44.676 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3005 ms) 2024-03-26T02:05:44.676 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-03-26T02:05:44.676 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (2934 ms) 2024-03-26T02:05:44.676 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-03-26T02:05:44.676 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3250 ms) 2024-03-26T02:05:44.676 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-03-26T02:05:44.676 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (2765 ms) 2024-03-26T02:05:44.676 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-03-26T02:05:44.676 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3274 ms) 2024-03-26T02:05:44.676 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-03-26T02:05:44.676 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (2761 ms) 2024-03-26T02:05:44.677 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-03-26T02:05:44.677 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (2953 ms) 2024-03-26T02:05:44.677 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-03-26T02:05:44.677 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (2998 ms) 2024-03-26T02:05:44.677 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-03-26T02:05:44.677 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3009 ms) 2024-03-26T02:05:44.677 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-03-26T02:05:44.677 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3034 ms) 2024-03-26T02:05:44.677 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-03-26T02:05:44.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:44 smithi072 ceph-mon[30177]: pgmap v226: 332 pgs: 40 unknown, 292 active+clean; 583 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:05:44.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:44 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi072-82404-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:44.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:44 smithi072 ceph-mon[30177]: osdmap e178: 8 total, 8 up, 8 in 2024-03-26T02:05:44.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:44 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3636275255' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:44.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:44 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:44.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:44 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:44.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:44 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:44.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:44 smithi072 ceph-mon[41619]: pgmap v226: 332 pgs: 40 unknown, 292 active+clean; 583 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:05:44.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:44 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi072-82404-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:44.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:44 smithi072 ceph-mon[41619]: osdmap e178: 8 total, 8 up, 8 in 2024-03-26T02:05:44.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:44 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3636275255' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:44.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:44 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:44.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:44 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:44.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:44 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:45.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:44 smithi173 ceph-mon[33839]: pgmap v226: 332 pgs: 40 unknown, 292 active+clean; 583 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:05:45.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:44 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi072-82404-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:45.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:44 smithi173 ceph-mon[33839]: osdmap e178: 8 total, 8 up, 8 in 2024-03-26T02:05:45.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:44 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3636275255' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:45.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:44 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:45.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:44 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:45.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:44 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:45.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:05:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:05:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:05:45.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi072-82387-29"}]': finished 2024-03-26T02:05:45.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:45.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[30177]: osdmap e179: 8 total, 8 up, 8 in 2024-03-26T02:05:45.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3636275255' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:45.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:45.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:05:45.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:05:45.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[30177]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:45.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi072-82387-29"}]': finished 2024-03-26T02:05:45.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:05:45.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[30177]: osdmap e180: 8 total, 8 up, 8 in 2024-03-26T02:05:45.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-17"}]: dispatch 2024-03-26T02:05:45.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2395025984' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi072-82404-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:45.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-17"}]: dispatch 2024-03-26T02:05:45.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi072-82404-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:45.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/541809219' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:45.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:45.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/541809219' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:45.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:45.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/541809219' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi072-82387-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:45.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi072-82387-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:45.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi072-82387-29"}]': finished 2024-03-26T02:05:45.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:45.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[41619]: osdmap e179: 8 total, 8 up, 8 in 2024-03-26T02:05:45.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3636275255' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:45.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:45.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:05:45.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:05:45.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[41619]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:45.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi072-82387-29"}]': finished 2024-03-26T02:05:45.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:05:45.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[41619]: osdmap e180: 8 total, 8 up, 8 in 2024-03-26T02:05:45.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-17"}]: dispatch 2024-03-26T02:05:45.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2395025984' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi072-82404-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:45.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-17"}]: dispatch 2024-03-26T02:05:45.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi072-82404-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:45.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/541809219' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:45.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:45.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/541809219' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:45.973 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:45.973 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/541809219' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi072-82387-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:45.973 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:45 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi072-82387-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:46.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:45 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi072-82387-29"}]': finished 2024-03-26T02:05:46.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:45 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:46.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:45 smithi173 ceph-mon[33839]: osdmap e179: 8 total, 8 up, 8 in 2024-03-26T02:05:46.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:45 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3636275255' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:46.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:45 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi072-82387-29"}]: dispatch 2024-03-26T02:05:46.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:45 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:05:46.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:45 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:05:46.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:45 smithi173 ceph-mon[33839]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:46.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:45 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi072-82387-29"}]': finished 2024-03-26T02:05:46.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:45 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:05:46.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:45 smithi173 ceph-mon[33839]: osdmap e180: 8 total, 8 up, 8 in 2024-03-26T02:05:46.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:45 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-17"}]: dispatch 2024-03-26T02:05:46.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:45 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2395025984' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi072-82404-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:46.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:45 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-17"}]: dispatch 2024-03-26T02:05:46.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:45 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi072-82404-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:46.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:45 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/541809219' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:46.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:45 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:46.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:45 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/541809219' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:46.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:45 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:46.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:45 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/541809219' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi072-82387-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:46.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:45 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi072-82387-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:46.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:46 smithi072 ceph-mon[30177]: pgmap v229: 324 pgs: 324 active+clean; 583 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:05:46.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:46 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-17"}]': finished 2024-03-26T02:05:46.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:46 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi072-82404-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:46.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:46 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi072-82387-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:46.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:46 smithi072 ceph-mon[30177]: osdmap e181: 8 total, 8 up, 8 in 2024-03-26T02:05:46.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:46 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-17", "mode": "writeback"}]: dispatch 2024-03-26T02:05:46.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:46 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/541809219' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi072-82387-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:46.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:46 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-17", "mode": "writeback"}]: dispatch 2024-03-26T02:05:46.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:46 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi072-82387-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:46.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:46 smithi072 ceph-mon[41619]: pgmap v229: 324 pgs: 324 active+clean; 583 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:05:46.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:46 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-17"}]': finished 2024-03-26T02:05:46.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:46 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi072-82404-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:46.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:46 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi072-82387-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:46.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:46 smithi072 ceph-mon[41619]: osdmap e181: 8 total, 8 up, 8 in 2024-03-26T02:05:46.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:46 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-17", "mode": "writeback"}]: dispatch 2024-03-26T02:05:46.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:46 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/541809219' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi072-82387-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:46.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:46 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-17", "mode": "writeback"}]: dispatch 2024-03-26T02:05:46.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:46 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi072-82387-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:47.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:46 smithi173 ceph-mon[33839]: pgmap v229: 324 pgs: 324 active+clean; 583 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:05:47.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:46 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-17"}]': finished 2024-03-26T02:05:47.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:46 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi072-82404-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:47.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:46 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi072-82387-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:47.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:46 smithi173 ceph-mon[33839]: osdmap e181: 8 total, 8 up, 8 in 2024-03-26T02:05:47.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:46 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-17", "mode": "writeback"}]: dispatch 2024-03-26T02:05:47.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:46 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/541809219' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi072-82387-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:47.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:46 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-17", "mode": "writeback"}]: dispatch 2024-03-26T02:05:47.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:46 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi072-82387-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:48.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:47 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:05:48.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:47 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-17", "mode": "writeback"}]': finished 2024-03-26T02:05:48.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:47 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:05:48.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:47 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-17", "mode": "writeback"}]': finished 2024-03-26T02:05:48.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:47 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:05:48.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:47 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-17", "mode": "writeback"}]': finished 2024-03-26T02:05:49.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:48 smithi072 ceph-mon[41619]: pgmap v232: 356 pgs: 32 unknown, 324 active+clean; 583 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:05:49.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:48 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:05:49.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:48 smithi072 ceph-mon[41619]: osdmap e182: 8 total, 8 up, 8 in 2024-03-26T02:05:49.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:48 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:05:49.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:48 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi072-82387-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi072-82387-30"}]': finished 2024-03-26T02:05:49.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:48 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:05:49.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:48 smithi072 ceph-mon[41619]: osdmap e183: 8 total, 8 up, 8 in 2024-03-26T02:05:49.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:48 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-17"}]: dispatch 2024-03-26T02:05:49.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:48 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3186506769' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi072-82404-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:49.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:48 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-17"}]: dispatch 2024-03-26T02:05:49.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:48 smithi072 ceph-mon[30177]: pgmap v232: 356 pgs: 32 unknown, 324 active+clean; 583 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:05:49.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:48 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:05:49.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:48 smithi072 ceph-mon[30177]: osdmap e182: 8 total, 8 up, 8 in 2024-03-26T02:05:49.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:48 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:05:49.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:48 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi072-82387-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi072-82387-30"}]': finished 2024-03-26T02:05:49.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:48 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:05:49.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:48 smithi072 ceph-mon[30177]: osdmap e183: 8 total, 8 up, 8 in 2024-03-26T02:05:49.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:48 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-17"}]: dispatch 2024-03-26T02:05:49.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:48 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3186506769' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi072-82404-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:49.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:48 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-17"}]: dispatch 2024-03-26T02:05:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:48 smithi173 ceph-mon[33839]: pgmap v232: 356 pgs: 32 unknown, 324 active+clean; 583 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:05:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:48 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:05:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:48 smithi173 ceph-mon[33839]: osdmap e182: 8 total, 8 up, 8 in 2024-03-26T02:05:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:48 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:05:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:48 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi072-82387-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi072-82387-30"}]': finished 2024-03-26T02:05:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:48 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:05:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:48 smithi173 ceph-mon[33839]: osdmap e183: 8 total, 8 up, 8 in 2024-03-26T02:05:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:48 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-17"}]: dispatch 2024-03-26T02:05:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:48 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3186506769' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi072-82404-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:48 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-17"}]: dispatch 2024-03-26T02:05:50.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:49 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:05:50.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:49 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3186506769' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi072-82404-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:50.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:49 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-17"}]': finished 2024-03-26T02:05:50.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:49 smithi072 ceph-mon[30177]: osdmap e184: 8 total, 8 up, 8 in 2024-03-26T02:05:50.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:49 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:05:50.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:49 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3186506769' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi072-82404-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:50.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:49 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-17"}]': finished 2024-03-26T02:05:50.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:49 smithi072 ceph-mon[41619]: osdmap e184: 8 total, 8 up, 8 in 2024-03-26T02:05:50.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:49 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:05:50.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:49 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3186506769' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi072-82404-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:50.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:49 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-17"}]': finished 2024-03-26T02:05:50.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:49 smithi173 ceph-mon[33839]: osdmap e184: 8 total, 8 up, 8 in 2024-03-26T02:05:51.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:05:50 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:05:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:51 smithi072 ceph-mon[30177]: pgmap v235: 364 pgs: 40 unknown, 324 active+clean; 583 KiB data, 592 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:05:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:51 smithi072 ceph-mon[30177]: osdmap e185: 8 total, 8 up, 8 in 2024-03-26T02:05:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:51 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/541809219' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:51 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:51.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:51 smithi072 ceph-mon[41619]: pgmap v235: 364 pgs: 40 unknown, 324 active+clean; 583 KiB data, 592 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:05:51.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:51 smithi072 ceph-mon[41619]: osdmap e185: 8 total, 8 up, 8 in 2024-03-26T02:05:51.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:51 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/541809219' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:51.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:51 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:51.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:51 smithi173 ceph-mon[33839]: pgmap v235: 364 pgs: 40 unknown, 324 active+clean; 583 KiB data, 592 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:05:51.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:51 smithi173 ceph-mon[33839]: osdmap e185: 8 total, 8 up, 8 in 2024-03-26T02:05:51.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:51 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/541809219' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:51.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:51 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:52.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:52 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:05:52.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:05:52.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:52 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi072-82387-30"}]': finished 2024-03-26T02:05:52.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:52 smithi072 ceph-mon[30177]: osdmap e186: 8 total, 8 up, 8 in 2024-03-26T02:05:52.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:52 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/541809219' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:52.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:52 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:52.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:52 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2530675017' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi072-82404-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:52.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:52 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:52.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:52 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:52.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:52 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi072-82404-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:52 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:05:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:05:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:52 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi072-82387-30"}]': finished 2024-03-26T02:05:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:52 smithi072 ceph-mon[41619]: osdmap e186: 8 total, 8 up, 8 in 2024-03-26T02:05:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:52 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/541809219' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:52 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:52.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:52 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2530675017' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi072-82404-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:52.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:52 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:52.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:52 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:52.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:52 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi072-82404-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:52.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:52 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:05:52.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:05:52.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:52 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi072-82387-30"}]': finished 2024-03-26T02:05:52.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:52 smithi173 ceph-mon[33839]: osdmap e186: 8 total, 8 up, 8 in 2024-03-26T02:05:52.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:52 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/541809219' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:52.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:52 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:52.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:52 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2530675017' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi072-82404-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:52.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:52 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi072-82387-30"}]: dispatch 2024-03-26T02:05:52.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:52 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:52.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:52 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi072-82404-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[30177]: pgmap v238: 292 pgs: 292 active+clean; 583 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:05:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi072-82387-30"}]': finished 2024-03-26T02:05:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi072-82404-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[30177]: osdmap e187: 8 total, 8 up, 8 in 2024-03-26T02:05:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:05:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:05:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3793885141' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3793885141' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:53.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3793885141' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi072-82387-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:53.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi072-82387-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:53.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[41619]: pgmap v238: 292 pgs: 292 active+clean; 583 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:05:53.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:53.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi072-82387-30"}]': finished 2024-03-26T02:05:53.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:53.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi072-82404-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:53.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[41619]: osdmap e187: 8 total, 8 up, 8 in 2024-03-26T02:05:53.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:05:53.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:05:53.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3793885141' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:53.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:53.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3793885141' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:53.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:53.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3793885141' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi072-82387-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:53.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:53 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi072-82387-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:53 smithi173 ceph-mon[33839]: pgmap v238: 292 pgs: 292 active+clean; 583 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:05:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:53 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:05:53.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:53 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi072-82387-30"}]': finished 2024-03-26T02:05:53.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:53 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:53.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:53 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi072-82404-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:53.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:53 smithi173 ceph-mon[33839]: osdmap e187: 8 total, 8 up, 8 in 2024-03-26T02:05:53.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:53 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:05:53.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:53 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:05:53.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:53 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3793885141' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:53.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:53 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:53.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:53 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3793885141' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:53.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:53 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:53.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:53 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3793885141' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi072-82387-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:53.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:53 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi072-82387-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:05:55.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:05:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:05:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:05:55.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:54 smithi072 ceph-mon[41619]: pgmap v241: 356 pgs: 64 unknown, 292 active+clean; 583 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:05:55.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:54 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:05:55.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:54 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi072-82387-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:55.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:54 smithi072 ceph-mon[41619]: osdmap e188: 8 total, 8 up, 8 in 2024-03-26T02:05:55.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:54 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-19"}]: dispatch 2024-03-26T02:05:55.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:54 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-19"}]: dispatch 2024-03-26T02:05:55.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:54 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3793885141' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi072-82387-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:55.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:54 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi072-82387-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:55.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:54 smithi072 ceph-mon[30177]: pgmap v241: 356 pgs: 64 unknown, 292 active+clean; 583 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:05:55.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:54 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:05:55.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:54 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi072-82387-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:55.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:54 smithi072 ceph-mon[30177]: osdmap e188: 8 total, 8 up, 8 in 2024-03-26T02:05:55.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:54 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-19"}]: dispatch 2024-03-26T02:05:55.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:54 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-19"}]: dispatch 2024-03-26T02:05:55.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:54 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3793885141' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi072-82387-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:55.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:54 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi072-82387-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:54 smithi173 ceph-mon[33839]: pgmap v241: 356 pgs: 64 unknown, 292 active+clean; 583 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:05:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:54 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:05:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:54 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi072-82387-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:05:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:54 smithi173 ceph-mon[33839]: osdmap e188: 8 total, 8 up, 8 in 2024-03-26T02:05:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:54 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-19"}]: dispatch 2024-03-26T02:05:55.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:54 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-19"}]: dispatch 2024-03-26T02:05:55.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:54 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3793885141' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi072-82387-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:55.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:54 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi072-82387-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:56.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:55 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-19"}]': finished 2024-03-26T02:05:56.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:55 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-19", "mode": "writeback"}]: dispatch 2024-03-26T02:05:56.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:55 smithi072 ceph-mon[30177]: osdmap e189: 8 total, 8 up, 8 in 2024-03-26T02:05:56.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:55 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-19", "mode": "writeback"}]: dispatch 2024-03-26T02:05:56.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:55 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2459358068' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi072-82404-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:56.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:55 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:05:56.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:55 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi072-82387-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi072-82387-31"}]': finished 2024-03-26T02:05:56.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:55 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-19", "mode": "writeback"}]': finished 2024-03-26T02:05:56.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:55 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2459358068' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi072-82404-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:56.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:55 smithi072 ceph-mon[30177]: osdmap e190: 8 total, 8 up, 8 in 2024-03-26T02:05:56.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:55 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:05:56.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:55 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:05:56.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:55 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-19"}]': finished 2024-03-26T02:05:56.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:55 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-19", "mode": "writeback"}]: dispatch 2024-03-26T02:05:56.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:55 smithi072 ceph-mon[41619]: osdmap e189: 8 total, 8 up, 8 in 2024-03-26T02:05:56.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:55 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-19", "mode": "writeback"}]: dispatch 2024-03-26T02:05:56.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:55 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2459358068' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi072-82404-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:56.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:55 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:05:56.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:55 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi072-82387-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi072-82387-31"}]': finished 2024-03-26T02:05:56.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:55 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-19", "mode": "writeback"}]': finished 2024-03-26T02:05:56.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:55 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2459358068' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi072-82404-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:56.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:55 smithi072 ceph-mon[41619]: osdmap e190: 8 total, 8 up, 8 in 2024-03-26T02:05:56.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:55 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:05:56.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:55 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:05:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:55 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-19"}]': finished 2024-03-26T02:05:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:55 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-19", "mode": "writeback"}]: dispatch 2024-03-26T02:05:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:55 smithi173 ceph-mon[33839]: osdmap e189: 8 total, 8 up, 8 in 2024-03-26T02:05:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:55 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-19", "mode": "writeback"}]: dispatch 2024-03-26T02:05:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:55 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2459358068' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi072-82404-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:55 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:05:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:55 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi072-82387-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi072-82387-31"}]': finished 2024-03-26T02:05:56.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:55 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-19", "mode": "writeback"}]': finished 2024-03-26T02:05:56.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:55 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2459358068' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi072-82404-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:05:56.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:55 smithi173 ceph-mon[33839]: osdmap e190: 8 total, 8 up, 8 in 2024-03-26T02:05:56.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:55 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:05:56.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:55 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:05:57.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[30177]: pgmap v244: 356 pgs: 32 creating+peering, 324 active+clean; 583 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:05:57.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2459358068' entity='client.admin' cmd=[{ 2024-03-26T02:05:57.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[30177]: "prefix": "osd pool set", 2024-03-26T02:05:57.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[30177]: "pool": "PoolEIOFlag_smithi072-82404-33", 2024-03-26T02:05:57.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[30177]: "var": "eio", 2024-03-26T02:05:57.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[30177]: "val": "true" 2024-03-26T02:05:57.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[30177]: }]: dispatch 2024-03-26T02:05:57.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:05:57.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2459358068' entity='client.admin' cmd='[{ 2024-03-26T02:05:57.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[30177]: "prefix": "osd pool set", 2024-03-26T02:05:57.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[30177]: "pool": "PoolEIOFlag_smithi072-82404-33", 2024-03-26T02:05:57.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[30177]: "var": "eio", 2024-03-26T02:05:57.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[30177]: "val": "true" 2024-03-26T02:05:57.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[30177]: }]': finished 2024-03-26T02:05:57.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[30177]: osdmap e191: 8 total, 8 up, 8 in 2024-03-26T02:05:57.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-19"}]: dispatch 2024-03-26T02:05:57.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-19"}]: dispatch 2024-03-26T02:05:57.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[41619]: pgmap v244: 356 pgs: 32 creating+peering, 324 active+clean; 583 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:05:57.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2459358068' entity='client.admin' cmd=[{ 2024-03-26T02:05:57.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[41619]: "prefix": "osd pool set", 2024-03-26T02:05:57.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[41619]: "pool": "PoolEIOFlag_smithi072-82404-33", 2024-03-26T02:05:57.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[41619]: "var": "eio", 2024-03-26T02:05:57.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[41619]: "val": "true" 2024-03-26T02:05:57.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[41619]: }]: dispatch 2024-03-26T02:05:57.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:05:57.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2459358068' entity='client.admin' cmd='[{ 2024-03-26T02:05:57.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[41619]: "prefix": "osd pool set", 2024-03-26T02:05:57.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[41619]: "pool": "PoolEIOFlag_smithi072-82404-33", 2024-03-26T02:05:57.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[41619]: "var": "eio", 2024-03-26T02:05:57.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[41619]: "val": "true" 2024-03-26T02:05:57.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[41619]: }]': finished 2024-03-26T02:05:57.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[41619]: osdmap e191: 8 total, 8 up, 8 in 2024-03-26T02:05:57.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-19"}]: dispatch 2024-03-26T02:05:57.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:56 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-19"}]: dispatch 2024-03-26T02:05:57.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:56 smithi173 ceph-mon[33839]: pgmap v244: 356 pgs: 32 creating+peering, 324 active+clean; 583 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:05:57.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:56 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2459358068' entity='client.admin' cmd=[{ 2024-03-26T02:05:57.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:56 smithi173 ceph-mon[33839]: "prefix": "osd pool set", 2024-03-26T02:05:57.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:56 smithi173 ceph-mon[33839]: "pool": "PoolEIOFlag_smithi072-82404-33", 2024-03-26T02:05:57.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:56 smithi173 ceph-mon[33839]: "var": "eio", 2024-03-26T02:05:57.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:56 smithi173 ceph-mon[33839]: "val": "true" 2024-03-26T02:05:57.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:56 smithi173 ceph-mon[33839]: }]: dispatch 2024-03-26T02:05:57.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:56 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:05:57.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:56 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2459358068' entity='client.admin' cmd='[{ 2024-03-26T02:05:57.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:56 smithi173 ceph-mon[33839]: "prefix": "osd pool set", 2024-03-26T02:05:57.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:56 smithi173 ceph-mon[33839]: "pool": "PoolEIOFlag_smithi072-82404-33", 2024-03-26T02:05:57.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:56 smithi173 ceph-mon[33839]: "var": "eio", 2024-03-26T02:05:57.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:56 smithi173 ceph-mon[33839]: "val": "true" 2024-03-26T02:05:57.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:56 smithi173 ceph-mon[33839]: }]': finished 2024-03-26T02:05:57.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:56 smithi173 ceph-mon[33839]: osdmap e191: 8 total, 8 up, 8 in 2024-03-26T02:05:57.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:56 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-19"}]: dispatch 2024-03-26T02:05:57.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:56 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-19"}]: dispatch 2024-03-26T02:05:58.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:57 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:05:58.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:57 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-19"}]': finished 2024-03-26T02:05:58.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:57 smithi072 ceph-mon[30177]: osdmap e192: 8 total, 8 up, 8 in 2024-03-26T02:05:58.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:57 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3793885141' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:58.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:57 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:58.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:57 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:05:58.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:57 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-19"}]': finished 2024-03-26T02:05:58.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:57 smithi072 ceph-mon[41619]: osdmap e192: 8 total, 8 up, 8 in 2024-03-26T02:05:58.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:57 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3793885141' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:58.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:57 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:58.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:57 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:05:58.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:57 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-19"}]': finished 2024-03-26T02:05:58.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:57 smithi173 ceph-mon[33839]: osdmap e192: 8 total, 8 up, 8 in 2024-03-26T02:05:58.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:57 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3793885141' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:58.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:57 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:59.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:58 smithi072 ceph-mon[30177]: pgmap v247: 364 pgs: 8 unknown, 32 creating+peering, 324 active+clean; 583 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:05:59.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:58 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi072-82387-31"}]': finished 2024-03-26T02:05:59.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:58 smithi072 ceph-mon[30177]: osdmap e193: 8 total, 8 up, 8 in 2024-03-26T02:05:59.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2521825480' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi072-82404-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:59.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3793885141' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:59.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:05:58 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:59.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:58 smithi072 ceph-mon[41619]: pgmap v247: 364 pgs: 8 unknown, 32 creating+peering, 324 active+clean; 583 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:05:59.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:58 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi072-82387-31"}]': finished 2024-03-26T02:05:59.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:58 smithi072 ceph-mon[41619]: osdmap e193: 8 total, 8 up, 8 in 2024-03-26T02:05:59.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2521825480' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi072-82404-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:59.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3793885141' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:59.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:05:58 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:59.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:58 smithi173 ceph-mon[33839]: pgmap v247: 364 pgs: 8 unknown, 32 creating+peering, 324 active+clean; 583 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:05:59.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:58 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi072-82387-31"}]': finished 2024-03-26T02:05:59.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:58 smithi173 ceph-mon[33839]: osdmap e193: 8 total, 8 up, 8 in 2024-03-26T02:05:59.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2521825480' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi072-82404-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:05:59.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3793885141' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:05:59.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:05:58 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi072-82387-31"}]: dispatch 2024-03-26T02:06:00.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:00 smithi072 ceph-mon[41619]: pgmap v250: 324 pgs: 32 unknown, 292 active+clean; 583 KiB data, 578 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:06:00.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2521825480' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi072-82404-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:00.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:00 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi072-82387-31"}]': finished 2024-03-26T02:06:00.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:00 smithi072 ceph-mon[41619]: osdmap e194: 8 total, 8 up, 8 in 2024-03-26T02:06:00.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:00.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:00 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:00.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/453558275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:00.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:00 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:00.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/453558275' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:00.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:00 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:00.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/453558275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi072-82387-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:00.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:00 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi072-82387-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:00.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:00 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:06:00.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:00 smithi072 ceph-mon[30177]: pgmap v250: 324 pgs: 32 unknown, 292 active+clean; 583 KiB data, 578 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:06:00.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2521825480' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi072-82404-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:00.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:00 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi072-82387-31"}]': finished 2024-03-26T02:06:00.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:00 smithi072 ceph-mon[30177]: osdmap e194: 8 total, 8 up, 8 in 2024-03-26T02:06:00.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:00.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:00 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:00.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/453558275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:00.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:00 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:00.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/453558275' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:00.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:00 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:00.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/453558275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi072-82387-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:00.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:00 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi072-82387-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:00.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:00 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:06:01.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:06:00 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:06:01.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:00 smithi173 ceph-mon[33839]: pgmap v250: 324 pgs: 32 unknown, 292 active+clean; 583 KiB data, 578 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:06:01.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2521825480' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi072-82404-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:01.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:00 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi072-82387-31"}]': finished 2024-03-26T02:06:01.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:00 smithi173 ceph-mon[33839]: osdmap e194: 8 total, 8 up, 8 in 2024-03-26T02:06:01.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:01.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:00 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:01.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/453558275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:01.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:00 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:01.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/453558275' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:01.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:00 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:01.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/453558275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi072-82387-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:01.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:00 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi072-82387-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:01.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:00 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:06:02.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:01 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:02.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:01 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi072-82387-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:02.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:01 smithi072 ceph-mon[41619]: osdmap e195: 8 total, 8 up, 8 in 2024-03-26T02:06:02.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/453558275' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi072-82387-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:02.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:01 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi072-82387-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:01 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:06:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:01 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:01 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:06:02.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:01 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:02.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:01 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi072-82387-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:02.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:01 smithi072 ceph-mon[30177]: osdmap e195: 8 total, 8 up, 8 in 2024-03-26T02:06:02.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/453558275' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi072-82387-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:02.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:01 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi072-82387-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:02.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:01 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:06:02.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:02.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:01 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:02.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:01 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:06:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:01 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:01 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi072-82387-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:01 smithi173 ceph-mon[33839]: osdmap e195: 8 total, 8 up, 8 in 2024-03-26T02:06:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/453558275' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi072-82387-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:01 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi072-82387-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:01 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:06:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:01 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:02.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:01 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:06:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:02 smithi072 ceph-mon[30177]: pgmap v253: 324 pgs: 32 creating+peering, 292 active+clean; 583 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:06:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-21"}]: dispatch 2024-03-26T02:06:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:02 smithi072 ceph-mon[30177]: osdmap e196: 8 total, 8 up, 8 in 2024-03-26T02:06:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3081463779' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi072-82404-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:02 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-21"}]: dispatch 2024-03-26T02:06:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:02 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi072-82387-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi072-82387-32"}]': finished 2024-03-26T02:06:03.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3081463779' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi072-82404-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:03.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:02 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-21"}]': finished 2024-03-26T02:06:03.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:02 smithi072 ceph-mon[30177]: osdmap e197: 8 total, 8 up, 8 in 2024-03-26T02:06:03.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-21", "mode": "writeback"}]: dispatch 2024-03-26T02:06:03.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:02 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-21", "mode": "writeback"}]: dispatch 2024-03-26T02:06:03.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:02 smithi072 ceph-mon[41619]: pgmap v253: 324 pgs: 32 creating+peering, 292 active+clean; 583 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:06:03.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-21"}]: dispatch 2024-03-26T02:06:03.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:02 smithi072 ceph-mon[41619]: osdmap e196: 8 total, 8 up, 8 in 2024-03-26T02:06:03.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3081463779' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi072-82404-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:03.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:02 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-21"}]: dispatch 2024-03-26T02:06:03.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:02 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi072-82387-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi072-82387-32"}]': finished 2024-03-26T02:06:03.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3081463779' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi072-82404-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:03.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:02 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-21"}]': finished 2024-03-26T02:06:03.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:02 smithi072 ceph-mon[41619]: osdmap e197: 8 total, 8 up, 8 in 2024-03-26T02:06:03.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-21", "mode": "writeback"}]: dispatch 2024-03-26T02:06:03.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:02 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-21", "mode": "writeback"}]: dispatch 2024-03-26T02:06:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:02 smithi173 ceph-mon[33839]: pgmap v253: 324 pgs: 32 creating+peering, 292 active+clean; 583 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:06:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-21"}]: dispatch 2024-03-26T02:06:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:02 smithi173 ceph-mon[33839]: osdmap e196: 8 total, 8 up, 8 in 2024-03-26T02:06:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3081463779' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi072-82404-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:02 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-21"}]: dispatch 2024-03-26T02:06:03.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:02 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi072-82387-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi072-82387-32"}]': finished 2024-03-26T02:06:03.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3081463779' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi072-82404-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:03.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:02 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-21"}]': finished 2024-03-26T02:06:03.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:02 smithi173 ceph-mon[33839]: osdmap e197: 8 total, 8 up, 8 in 2024-03-26T02:06:03.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-21", "mode": "writeback"}]: dispatch 2024-03-26T02:06:03.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:02 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-21", "mode": "writeback"}]: dispatch 2024-03-26T02:06:04.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:03 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:06:04.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:03 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-21", "mode": "writeback"}]': finished 2024-03-26T02:06:04.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:03 smithi072 ceph-mon[30177]: osdmap e198: 8 total, 8 up, 8 in 2024-03-26T02:06:04.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:03 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:06:04.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:03 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-21", "mode": "writeback"}]': finished 2024-03-26T02:06:04.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:03 smithi072 ceph-mon[41619]: osdmap e198: 8 total, 8 up, 8 in 2024-03-26T02:06:04.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:03 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:06:04.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:03 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-21", "mode": "writeback"}]': finished 2024-03-26T02:06:04.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:03 smithi173 ceph-mon[33839]: osdmap e198: 8 total, 8 up, 8 in 2024-03-26T02:06:05.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:06:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:06:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:06:05.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:04 smithi072 ceph-mon[41619]: pgmap v256: 364 pgs: 40 unknown, 32 creating+peering, 292 active+clean; 583 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:06:05.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:04 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:05.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:04 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:05.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:04 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:06:05.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:04 smithi072 ceph-mon[41619]: osdmap e199: 8 total, 8 up, 8 in 2024-03-26T02:06:05.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:04 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-21"}]: dispatch 2024-03-26T02:06:05.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:04 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-21"}]: dispatch 2024-03-26T02:06:05.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:04 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/453558275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:05.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:04 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:05.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:04 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2409073333' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:05.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:04 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:05.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:04 smithi072 ceph-mon[30177]: pgmap v256: 364 pgs: 40 unknown, 32 creating+peering, 292 active+clean; 583 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:06:05.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:04 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:05.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:04 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:05.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:04 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:06:05.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:04 smithi072 ceph-mon[30177]: osdmap e199: 8 total, 8 up, 8 in 2024-03-26T02:06:05.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:04 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-21"}]: dispatch 2024-03-26T02:06:05.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:04 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-21"}]: dispatch 2024-03-26T02:06:05.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:04 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/453558275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:05.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:04 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:05.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:04 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2409073333' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:05.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:04 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:05.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:04 smithi173 ceph-mon[33839]: pgmap v256: 364 pgs: 40 unknown, 32 creating+peering, 292 active+clean; 583 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:06:05.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:04 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:05.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:04 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:05.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:04 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:06:05.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:04 smithi173 ceph-mon[33839]: osdmap e199: 8 total, 8 up, 8 in 2024-03-26T02:06:05.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:04 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-21"}]: dispatch 2024-03-26T02:06:05.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:04 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-21"}]: dispatch 2024-03-26T02:06:05.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:04 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/453558275' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:05.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:04 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:05.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:04 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2409073333' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:05.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:04 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:06.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:05 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:06:06.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-21"}]': finished 2024-03-26T02:06:06.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi072-82387-32"}]': finished 2024-03-26T02:06:06.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:06.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:05 smithi072 ceph-mon[30177]: osdmap e200: 8 total, 8 up, 8 in 2024-03-26T02:06:06.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/453558275' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:06.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:06.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:05 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:06:06.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-21"}]': finished 2024-03-26T02:06:06.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi072-82387-32"}]': finished 2024-03-26T02:06:06.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:06.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:05 smithi072 ceph-mon[41619]: osdmap e200: 8 total, 8 up, 8 in 2024-03-26T02:06:06.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/453558275' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:06.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:06.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:05 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:06:06.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-21"}]': finished 2024-03-26T02:06:06.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi072-82387-32"}]': finished 2024-03-26T02:06:06.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:06.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:05 smithi173 ceph-mon[33839]: osdmap e200: 8 total, 8 up, 8 in 2024-03-26T02:06:06.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/453558275' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:06.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi072-82387-32"}]: dispatch 2024-03-26T02:06:07.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:06 smithi072 ceph-mon[30177]: pgmap v259: 356 pgs: 32 creating+peering, 324 active+clean; 583 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T02:06:07.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:06 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:06:07.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:06 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi072-82387-32"}]': finished 2024-03-26T02:06:07.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:06 smithi072 ceph-mon[30177]: osdmap e201: 8 total, 8 up, 8 in 2024-03-26T02:06:07.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1103791247' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:07.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2498920509' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:07.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:06 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:07.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:06 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:07.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2498920509' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:07.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:06 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:07.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2498920509' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi072-82387-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:07.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:06 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi072-82387-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:07.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:06 smithi072 ceph-mon[41619]: pgmap v259: 356 pgs: 32 creating+peering, 324 active+clean; 583 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T02:06:07.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:06 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:06:07.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:06 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi072-82387-32"}]': finished 2024-03-26T02:06:07.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:06 smithi072 ceph-mon[41619]: osdmap e201: 8 total, 8 up, 8 in 2024-03-26T02:06:07.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1103791247' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:07.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2498920509' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:07.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:06 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:07.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:06 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:07.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2498920509' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:07.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:06 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:07.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2498920509' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi072-82387-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:07.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:06 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi072-82387-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:07.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:06 smithi173 ceph-mon[33839]: pgmap v259: 356 pgs: 32 creating+peering, 324 active+clean; 583 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T02:06:07.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:06 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:06:07.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:06 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi072-82387-32"}]': finished 2024-03-26T02:06:07.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:06 smithi173 ceph-mon[33839]: osdmap e201: 8 total, 8 up, 8 in 2024-03-26T02:06:07.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1103791247' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:07.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2498920509' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:07.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:06 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:07.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:06 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:07.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2498920509' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:07.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:06 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:07.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2498920509' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi072-82387-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:07.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:06 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi072-82387-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:08 smithi072 ceph-mon[30177]: pgmap v262: 356 pgs: 32 unknown, 32 creating+peering, 292 active+clean; 583 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T02:06:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi072-82387-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:08 smithi072 ceph-mon[30177]: osdmap e202: 8 total, 8 up, 8 in 2024-03-26T02:06:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2498920509' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi072-82387-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi072-82387-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:09.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:09.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:08 smithi072 ceph-mon[41619]: pgmap v262: 356 pgs: 32 unknown, 32 creating+peering, 292 active+clean; 583 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T02:06:09.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:09.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi072-82387-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:09.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:08 smithi072 ceph-mon[41619]: osdmap e202: 8 total, 8 up, 8 in 2024-03-26T02:06:09.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2498920509' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi072-82387-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:09.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi072-82387-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:09.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:09.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:08 smithi173 ceph-mon[33839]: pgmap v262: 356 pgs: 32 unknown, 32 creating+peering, 292 active+clean; 583 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T02:06:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi072-82387-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:08 smithi173 ceph-mon[33839]: osdmap e202: 8 total, 8 up, 8 in 2024-03-26T02:06:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2498920509' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi072-82387-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi072-82387-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:09.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:09.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:10.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:10.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:09 smithi173 ceph-mon[33839]: osdmap e203: 8 total, 8 up, 8 in 2024-03-26T02:06:10.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2798561118' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:10.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:10.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi072-82387-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi072-82387-33"}]': finished 2024-03-26T02:06:10.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:10.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:09 smithi173 ceph-mon[33839]: osdmap e204: 8 total, 8 up, 8 in 2024-03-26T02:06:10.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:10.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:10.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:10.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:09 smithi072 ceph-mon[30177]: osdmap e203: 8 total, 8 up, 8 in 2024-03-26T02:06:10.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2798561118' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:10.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:10.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi072-82387-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi072-82387-33"}]': finished 2024-03-26T02:06:10.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:10.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:09 smithi072 ceph-mon[30177]: osdmap e204: 8 total, 8 up, 8 in 2024-03-26T02:06:10.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:10.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:10.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:10.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:09 smithi072 ceph-mon[41619]: osdmap e203: 8 total, 8 up, 8 in 2024-03-26T02:06:10.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2798561118' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:10.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:10.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi072-82387-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi072-82387-33"}]': finished 2024-03-26T02:06:10.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi072-82404-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:10.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:09 smithi072 ceph-mon[41619]: osdmap e204: 8 total, 8 up, 8 in 2024-03-26T02:06:10.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:10.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:10.930 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:06:10 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:06:11.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:10 smithi072 ceph-mon[41619]: pgmap v265: 420 pgs: 96 unknown, 32 creating+peering, 292 active+clean; 583 KiB data, 583 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:06:11.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:06:11.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:10 smithi072 ceph-mon[41619]: osdmap e205: 8 total, 8 up, 8 in 2024-03-26T02:06:11.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-23"}]: dispatch 2024-03-26T02:06:11.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-23"}]: dispatch 2024-03-26T02:06:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:10 smithi072 ceph-mon[30177]: pgmap v265: 420 pgs: 96 unknown, 32 creating+peering, 292 active+clean; 583 KiB data, 583 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:06:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:06:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:10 smithi072 ceph-mon[30177]: osdmap e205: 8 total, 8 up, 8 in 2024-03-26T02:06:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-23"}]: dispatch 2024-03-26T02:06:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-23"}]: dispatch 2024-03-26T02:06:11.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:10 smithi173 ceph-mon[33839]: pgmap v265: 420 pgs: 96 unknown, 32 creating+peering, 292 active+clean; 583 KiB data, 583 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:06:11.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:06:11.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:10 smithi173 ceph-mon[33839]: osdmap e205: 8 total, 8 up, 8 in 2024-03-26T02:06:11.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-23"}]: dispatch 2024-03-26T02:06:11.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-23"}]: dispatch 2024-03-26T02:06:12.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:11 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:06:12.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:11 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-23"}]': finished 2024-03-26T02:06:12.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:11 smithi072 ceph-mon[30177]: osdmap e206: 8 total, 8 up, 8 in 2024-03-26T02:06:12.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:11 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-23", "mode": "writeback"}]: dispatch 2024-03-26T02:06:12.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:11 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2498920509' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:12.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:11 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-23", "mode": "writeback"}]: dispatch 2024-03-26T02:06:12.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:11 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:11 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:06:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:11 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-23"}]': finished 2024-03-26T02:06:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:11 smithi072 ceph-mon[41619]: osdmap e206: 8 total, 8 up, 8 in 2024-03-26T02:06:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:11 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-23", "mode": "writeback"}]: dispatch 2024-03-26T02:06:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:11 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2498920509' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:11 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-23", "mode": "writeback"}]: dispatch 2024-03-26T02:06:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:11 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:11 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:06:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:11 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-23"}]': finished 2024-03-26T02:06:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:11 smithi173 ceph-mon[33839]: osdmap e206: 8 total, 8 up, 8 in 2024-03-26T02:06:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:11 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-23", "mode": "writeback"}]: dispatch 2024-03-26T02:06:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:11 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2498920509' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:11 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-23", "mode": "writeback"}]: dispatch 2024-03-26T02:06:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:11 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:13.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:12 smithi072 ceph-mon[30177]: pgmap v268: 396 pgs: 8 creating+peering, 388 active+clean; 584 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-26T02:06:13.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:12 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:06:13.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-23", "mode": "writeback"}]': finished 2024-03-26T02:06:13.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi072-82387-33"}]': finished 2024-03-26T02:06:13.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:12 smithi072 ceph-mon[30177]: osdmap e207: 8 total, 8 up, 8 in 2024-03-26T02:06:13.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2498920509' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:13.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:13.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:12 smithi072 ceph-mon[41619]: pgmap v268: 396 pgs: 8 creating+peering, 388 active+clean; 584 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-26T02:06:13.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:12 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:06:13.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-23", "mode": "writeback"}]': finished 2024-03-26T02:06:13.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi072-82387-33"}]': finished 2024-03-26T02:06:13.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:12 smithi072 ceph-mon[41619]: osdmap e207: 8 total, 8 up, 8 in 2024-03-26T02:06:13.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2498920509' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:13.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:12 smithi173 ceph-mon[33839]: pgmap v268: 396 pgs: 8 creating+peering, 388 active+clean; 584 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-26T02:06:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:12 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:06:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-23", "mode": "writeback"}]': finished 2024-03-26T02:06:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi072-82387-33"}]': finished 2024-03-26T02:06:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:12 smithi173 ceph-mon[33839]: osdmap e207: 8 total, 8 up, 8 in 2024-03-26T02:06:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2498920509' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi072-82387-33"}]: dispatch 2024-03-26T02:06:14.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:14 smithi072 ceph-mon[30177]: pgmap v271: 324 pgs: 324 active+clean; 584 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T02:06:14.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:14 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi072-82387-33"}]': finished 2024-03-26T02:06:14.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:14 smithi072 ceph-mon[30177]: osdmap e208: 8 total, 8 up, 8 in 2024-03-26T02:06:14.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:14 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1885833793' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:14.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:14 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:14.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:14 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi072-82387-34"}]: dispatch 2024-03-26T02:06:14.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:14 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi072-82387-34"}]: dispatch 2024-03-26T02:06:14.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:14 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi072-82387-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:14.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:14 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:14.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:14 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:14.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:14 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:06:14.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:14 smithi072 ceph-mon[41619]: pgmap v271: 324 pgs: 324 active+clean; 584 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T02:06:14.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:14 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi072-82387-33"}]': finished 2024-03-26T02:06:14.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:14 smithi072 ceph-mon[41619]: osdmap e208: 8 total, 8 up, 8 in 2024-03-26T02:06:14.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:14 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1885833793' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:14.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:14 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:14.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:14 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi072-82387-34"}]: dispatch 2024-03-26T02:06:14.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:14 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi072-82387-34"}]: dispatch 2024-03-26T02:06:14.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:14 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi072-82387-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:14.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:14 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:14.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:14 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:14.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:14 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:06:15.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:14 smithi173 ceph-mon[33839]: pgmap v271: 324 pgs: 324 active+clean; 584 KiB data, 592 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T02:06:15.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:14 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi072-82387-33"}]': finished 2024-03-26T02:06:15.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:14 smithi173 ceph-mon[33839]: osdmap e208: 8 total, 8 up, 8 in 2024-03-26T02:06:15.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:14 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1885833793' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:15.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:14 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:15.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:14 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi072-82387-34"}]: dispatch 2024-03-26T02:06:15.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:14 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi072-82387-34"}]: dispatch 2024-03-26T02:06:15.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:14 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi072-82387-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:15.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:14 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:15.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:14 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:15.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:14 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:06:15.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:06:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:06:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:06:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi072-82387-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:06:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:15 smithi072 ceph-mon[30177]: osdmap e209: 8 total, 8 up, 8 in 2024-03-26T02:06:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi072-82387-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi072-82387-34"}]: dispatch 2024-03-26T02:06:16.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-23"}]: dispatch 2024-03-26T02:06:16.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-23"}]: dispatch 2024-03-26T02:06:16.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:15 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:06:16.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-23"}]': finished 2024-03-26T02:06:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi072-82387-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:06:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:15 smithi072 ceph-mon[41619]: osdmap e209: 8 total, 8 up, 8 in 2024-03-26T02:06:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi072-82387-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi072-82387-34"}]: dispatch 2024-03-26T02:06:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-23"}]: dispatch 2024-03-26T02:06:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-23"}]: dispatch 2024-03-26T02:06:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:15 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:06:16.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-23"}]': finished 2024-03-26T02:06:16.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:16.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi072-82387-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:16.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:06:16.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:15 smithi173 ceph-mon[33839]: osdmap e209: 8 total, 8 up, 8 in 2024-03-26T02:06:16.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi072-82387-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi072-82387-34"}]: dispatch 2024-03-26T02:06:16.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-23"}]: dispatch 2024-03-26T02:06:16.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-23"}]: dispatch 2024-03-26T02:06:16.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:15 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:06:16.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-23"}]': finished 2024-03-26T02:06:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:16 smithi072 ceph-mon[30177]: pgmap v274: 356 pgs: 356 active+clean; 584 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-26T02:06:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:16 smithi072 ceph-mon[30177]: osdmap e210: 8 total, 8 up, 8 in 2024-03-26T02:06:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:16 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447575981' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:16 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:16 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi072-82387-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi072-82387-34"}]': finished 2024-03-26T02:06:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:16 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:16 smithi072 ceph-mon[30177]: osdmap e211: 8 total, 8 up, 8 in 2024-03-26T02:06:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:16 smithi072 ceph-mon[41619]: pgmap v274: 356 pgs: 356 active+clean; 584 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-26T02:06:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:16 smithi072 ceph-mon[41619]: osdmap e210: 8 total, 8 up, 8 in 2024-03-26T02:06:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:16 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447575981' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:16 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:16 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi072-82387-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi072-82387-34"}]': finished 2024-03-26T02:06:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:16 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:16 smithi072 ceph-mon[41619]: osdmap e211: 8 total, 8 up, 8 in 2024-03-26T02:06:17.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:16 smithi173 ceph-mon[33839]: pgmap v274: 356 pgs: 356 active+clean; 584 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 0 B/s wr, 4 op/s 2024-03-26T02:06:17.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:16 smithi173 ceph-mon[33839]: osdmap e210: 8 total, 8 up, 8 in 2024-03-26T02:06:17.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:16 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447575981' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:17.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:16 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:17.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:16 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi072-82387-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi072-82387-34"}]': finished 2024-03-26T02:06:17.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:16 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:17.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:16 smithi173 ceph-mon[33839]: osdmap e211: 8 total, 8 up, 8 in 2024-03-26T02:06:18.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:18 smithi072 ceph-mon[30177]: pgmap v277: 364 pgs: 40 unknown, 324 active+clean; 584 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T02:06:18.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:18 smithi072 ceph-mon[30177]: osdmap e212: 8 total, 8 up, 8 in 2024-03-26T02:06:18.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:18.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3274828590' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:18.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:18.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:18.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:18 smithi072 ceph-mon[41619]: pgmap v277: 364 pgs: 40 unknown, 324 active+clean; 584 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T02:06:18.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:18 smithi072 ceph-mon[41619]: osdmap e212: 8 total, 8 up, 8 in 2024-03-26T02:06:18.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:18.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3274828590' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:18.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:18.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:19.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:18 smithi173 ceph-mon[33839]: pgmap v277: 364 pgs: 40 unknown, 324 active+clean; 584 KiB data, 585 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T02:06:19.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:18 smithi173 ceph-mon[33839]: osdmap e212: 8 total, 8 up, 8 in 2024-03-26T02:06:19.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:19.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3274828590' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:19.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:19.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:19.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:19 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:19.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:19 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:19.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:19 smithi072 ceph-mon[30177]: osdmap e213: 8 total, 8 up, 8 in 2024-03-26T02:06:19.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi072-82387-34"}]: dispatch 2024-03-26T02:06:19.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi072-82387-34"}]': finished 2024-03-26T02:06:19.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:19 smithi072 ceph-mon[30177]: osdmap e214: 8 total, 8 up, 8 in 2024-03-26T02:06:19.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi072-82387-34"}]: dispatch 2024-03-26T02:06:19.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3604967708' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:19.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:19 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:19.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:19 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:19.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:19 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:19.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:19 smithi072 ceph-mon[41619]: osdmap e213: 8 total, 8 up, 8 in 2024-03-26T02:06:19.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi072-82387-34"}]: dispatch 2024-03-26T02:06:19.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi072-82387-34"}]': finished 2024-03-26T02:06:19.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:19 smithi072 ceph-mon[41619]: osdmap e214: 8 total, 8 up, 8 in 2024-03-26T02:06:19.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi072-82387-34"}]: dispatch 2024-03-26T02:06:19.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3604967708' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:19.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:19 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:20.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:19 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:20.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:19 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:20.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:19 smithi173 ceph-mon[33839]: osdmap e213: 8 total, 8 up, 8 in 2024-03-26T02:06:20.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi072-82387-34"}]: dispatch 2024-03-26T02:06:20.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi072-82387-34"}]': finished 2024-03-26T02:06:20.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:19 smithi173 ceph-mon[33839]: osdmap e214: 8 total, 8 up, 8 in 2024-03-26T02:06:20.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi072-82387-34"}]: dispatch 2024-03-26T02:06:20.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3604967708' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:20.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:19 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:21.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:06:20 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:06:21.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:20 smithi173 ceph-mon[33839]: pgmap v280: 420 pgs: 96 unknown, 324 active+clean; 584 KiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:06:21.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:20 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:06:21.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:21.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:20 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:21.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi072-82387-34"}]': finished 2024-03-26T02:06:21.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:20 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:21.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:20 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:06:21.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:20 smithi173 ceph-mon[33839]: osdmap e215: 8 total, 8 up, 8 in 2024-03-26T02:06:21.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-25"}]: dispatch 2024-03-26T02:06:21.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:20 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-25"}]: dispatch 2024-03-26T02:06:21.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1620251010' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:21.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:20 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:21.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1620251010' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:21.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:20 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:21.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1620251010' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi072-82387-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:21.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:20 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi072-82387-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:21.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[30177]: pgmap v280: 420 pgs: 96 unknown, 324 active+clean; 584 KiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:06:21.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:06:21.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:21.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:21.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi072-82387-34"}]': finished 2024-03-26T02:06:21.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:21.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:06:21.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[30177]: osdmap e215: 8 total, 8 up, 8 in 2024-03-26T02:06:21.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-25"}]: dispatch 2024-03-26T02:06:21.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-25"}]: dispatch 2024-03-26T02:06:21.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1620251010' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:21.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:21.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1620251010' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:21.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:21.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1620251010' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi072-82387-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:21.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi072-82387-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:21.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[41619]: pgmap v280: 420 pgs: 96 unknown, 324 active+clean; 584 KiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:06:21.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:06:21.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:21.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:21.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2223638274' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi072-82387-34"}]': finished 2024-03-26T02:06:21.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:21.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:06:21.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[41619]: osdmap e215: 8 total, 8 up, 8 in 2024-03-26T02:06:21.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-25"}]: dispatch 2024-03-26T02:06:21.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-25"}]: dispatch 2024-03-26T02:06:21.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1620251010' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:21.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:21.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1620251010' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:21.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:21.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1620251010' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi072-82387-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:21.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:20 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi072-82387-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:22.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:21 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:06:22.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:21 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:06:22.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:21 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-25"}]': finished 2024-03-26T02:06:22.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:21 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi072-82387-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:22.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1620251010' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi072-82387-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:22.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:21 smithi072 ceph-mon[30177]: osdmap e216: 8 total, 8 up, 8 in 2024-03-26T02:06:22.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-25", "mode": "writeback"}]: dispatch 2024-03-26T02:06:22.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/502133004' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:22.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:21 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi072-82387-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:22.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:21 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-25", "mode": "writeback"}]: dispatch 2024-03-26T02:06:22.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:21 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:06:22.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:21 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:06:22.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:21 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-25"}]': finished 2024-03-26T02:06:22.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:21 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi072-82387-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:22.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1620251010' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi072-82387-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:22.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:21 smithi072 ceph-mon[41619]: osdmap e216: 8 total, 8 up, 8 in 2024-03-26T02:06:22.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-25", "mode": "writeback"}]: dispatch 2024-03-26T02:06:22.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/502133004' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:22.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:21 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi072-82387-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:22.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:21 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-25", "mode": "writeback"}]: dispatch 2024-03-26T02:06:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:21 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:06:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:21 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:06:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:21 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-25"}]': finished 2024-03-26T02:06:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:21 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi072-82387-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:22.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1620251010' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi072-82387-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:22.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:21 smithi173 ceph-mon[33839]: osdmap e216: 8 total, 8 up, 8 in 2024-03-26T02:06:22.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-25", "mode": "writeback"}]: dispatch 2024-03-26T02:06:22.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/502133004' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:22.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:21 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi072-82387-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:22.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:21 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-25", "mode": "writeback"}]: dispatch 2024-03-26T02:06:23.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:22 smithi072 ceph-mon[30177]: pgmap v283: 452 pgs: 32 creating+peering, 420 active+clean; 584 KiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T02:06:23.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:22 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:06:23.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:22 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/502133004' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:23.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:22 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-25", "mode": "writeback"}]': finished 2024-03-26T02:06:23.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:22 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:23.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:22 smithi072 ceph-mon[30177]: osdmap e217: 8 total, 8 up, 8 in 2024-03-26T02:06:23.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:22 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:23.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:22 smithi072 ceph-mon[41619]: pgmap v283: 452 pgs: 32 creating+peering, 420 active+clean; 584 KiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T02:06:23.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:22 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:06:23.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:22 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/502133004' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:23.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:22 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-25", "mode": "writeback"}]': finished 2024-03-26T02:06:23.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:22 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:23.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:22 smithi072 ceph-mon[41619]: osdmap e217: 8 total, 8 up, 8 in 2024-03-26T02:06:23.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:22 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:22 smithi173 ceph-mon[33839]: pgmap v283: 452 pgs: 32 creating+peering, 420 active+clean; 584 KiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T02:06:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:22 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:06:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:22 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/502133004' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi072-82404-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:22 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-25", "mode": "writeback"}]': finished 2024-03-26T02:06:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:22 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:22 smithi173 ceph-mon[33839]: osdmap e217: 8 total, 8 up, 8 in 2024-03-26T02:06:23.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:22 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:24.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:24 smithi072 ceph-mon[30177]: pgmap v286: 484 pgs: 32 unknown, 32 creating+peering, 420 active+clean; 584 KiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T02:06:24.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:24 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi072-82387-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi072-82387-35"}]': finished 2024-03-26T02:06:24.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:24 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:06:24.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:24 smithi072 ceph-mon[30177]: osdmap e218: 8 total, 8 up, 8 in 2024-03-26T02:06:24.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:24 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-25"}]: dispatch 2024-03-26T02:06:24.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:24 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-25"}]: dispatch 2024-03-26T02:06:24.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:24 smithi072 ceph-mon[41619]: pgmap v286: 484 pgs: 32 unknown, 32 creating+peering, 420 active+clean; 584 KiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T02:06:24.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:24 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi072-82387-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi072-82387-35"}]': finished 2024-03-26T02:06:24.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:24 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:06:24.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:24 smithi072 ceph-mon[41619]: osdmap e218: 8 total, 8 up, 8 in 2024-03-26T02:06:24.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:24 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-25"}]: dispatch 2024-03-26T02:06:24.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:24 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-25"}]: dispatch 2024-03-26T02:06:25.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:24 smithi173 ceph-mon[33839]: pgmap v286: 484 pgs: 32 unknown, 32 creating+peering, 420 active+clean; 584 KiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T02:06:25.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:24 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi072-82387-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi072-82387-35"}]': finished 2024-03-26T02:06:25.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:24 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:06:25.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:24 smithi173 ceph-mon[33839]: osdmap e218: 8 total, 8 up, 8 in 2024-03-26T02:06:25.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:24 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-25"}]: dispatch 2024-03-26T02:06:25.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:24 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-25"}]: dispatch 2024-03-26T02:06:25.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:06:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:06:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:06:25.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:25 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:06:25.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:25 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-25"}]': finished 2024-03-26T02:06:25.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:25 smithi072 ceph-mon[30177]: osdmap e219: 8 total, 8 up, 8 in 2024-03-26T02:06:25.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:25 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:06:25.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:25 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-25"}]': finished 2024-03-26T02:06:25.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:25 smithi072 ceph-mon[41619]: osdmap e219: 8 total, 8 up, 8 in 2024-03-26T02:06:26.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:25 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:06:26.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:25 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-25"}]': finished 2024-03-26T02:06:26.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:25 smithi173 ceph-mon[33839]: osdmap e219: 8 total, 8 up, 8 in 2024-03-26T02:06:26.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:26 smithi072 ceph-mon[30177]: pgmap v289: 428 pgs: 428 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T02:06:26.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:26 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:06:26.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:26 smithi072 ceph-mon[30177]: osdmap e220: 8 total, 8 up, 8 in 2024-03-26T02:06:26.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:26 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1620251010' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:26.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:26 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:26 smithi072 ceph-mon[41619]: pgmap v289: 428 pgs: 428 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T02:06:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:26 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:06:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:26 smithi072 ceph-mon[41619]: osdmap e220: 8 total, 8 up, 8 in 2024-03-26T02:06:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:26 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1620251010' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:26 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:26 smithi173 ceph-mon[33839]: pgmap v289: 428 pgs: 428 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T02:06:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:26 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:06:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:26 smithi173 ceph-mon[33839]: osdmap e220: 8 total, 8 up, 8 in 2024-03-26T02:06:27.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:26 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1620251010' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:27.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:26 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:27.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:27 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi072-82387-35"}]': finished 2024-03-26T02:06:27.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:27 smithi072 ceph-mon[30177]: osdmap e221: 8 total, 8 up, 8 in 2024-03-26T02:06:27.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:27 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:27.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:27 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1620251010' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:27.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:27 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:27.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:27 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:27.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:27 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi072-82387-35"}]': finished 2024-03-26T02:06:27.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:27 smithi072 ceph-mon[41619]: osdmap e221: 8 total, 8 up, 8 in 2024-03-26T02:06:27.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:27 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:27.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:27 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1620251010' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:27.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:27 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:27.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:27 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:28.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:27 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi072-82387-35"}]': finished 2024-03-26T02:06:28.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:27 smithi173 ceph-mon[33839]: osdmap e221: 8 total, 8 up, 8 in 2024-03-26T02:06:28.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:27 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:28.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:27 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1620251010' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:28.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:27 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:28.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:27 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi072-82387-35"}]: dispatch 2024-03-26T02:06:28.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:28 smithi072 ceph-mon[30177]: pgmap v292: 356 pgs: 32 unknown, 324 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:06:28.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:28 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:28.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:28 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi072-82387-35"}]': finished 2024-03-26T02:06:28.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:28 smithi072 ceph-mon[30177]: osdmap e222: 8 total, 8 up, 8 in 2024-03-26T02:06:28.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:28 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2918319052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:28.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:28 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:28.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:28 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2918319052' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:28.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:28 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:28.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:28 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2918319052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi072-82387-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:28.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:28 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi072-82387-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:28.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:28 smithi072 ceph-mon[41619]: pgmap v292: 356 pgs: 32 unknown, 324 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:06:28.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:28 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:28.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:28 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi072-82387-35"}]': finished 2024-03-26T02:06:28.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:28 smithi072 ceph-mon[41619]: osdmap e222: 8 total, 8 up, 8 in 2024-03-26T02:06:28.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:28 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2918319052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:28.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:28 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:28.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:28 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2918319052' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:28.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:28 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:28.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:28 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2918319052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi072-82387-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:28.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:28 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi072-82387-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:29.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:28 smithi173 ceph-mon[33839]: pgmap v292: 356 pgs: 32 unknown, 324 active+clean; 584 KiB data, 562 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:06:29.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:28 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:29.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:28 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi072-82387-35"}]': finished 2024-03-26T02:06:29.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:28 smithi173 ceph-mon[33839]: osdmap e222: 8 total, 8 up, 8 in 2024-03-26T02:06:29.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:28 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2918319052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:29.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:28 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:29.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:28 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2918319052' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:29.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:28 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:29.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:28 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2918319052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi072-82387-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:29.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:28 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi072-82387-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:29.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:29 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi072-82387-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:29.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:29 smithi072 ceph-mon[30177]: osdmap e223: 8 total, 8 up, 8 in 2024-03-26T02:06:29.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:29 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2918319052' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi072-82387-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:29.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:29 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi072-82387-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:29.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:29 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1440026561' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi072-82404-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:29.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:29 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi072-82404-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:29.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:29 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:29.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:29 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:29.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:29 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi072-82387-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:29.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:29 smithi072 ceph-mon[41619]: osdmap e223: 8 total, 8 up, 8 in 2024-03-26T02:06:29.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:29 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2918319052' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi072-82387-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:29.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:29 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi072-82387-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:29.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:29 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1440026561' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi072-82404-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:29.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:29 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi072-82404-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:29.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:29 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:29.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:29 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:30.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:29 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi072-82387-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:30.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:29 smithi173 ceph-mon[33839]: osdmap e223: 8 total, 8 up, 8 in 2024-03-26T02:06:30.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:29 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2918319052' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi072-82387-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:30.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:29 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi072-82387-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:30.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:29 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1440026561' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi072-82404-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:30.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:29 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi072-82404-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:30.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:29 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:30.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:29 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:30.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:30 smithi072 ceph-mon[30177]: pgmap v295: 356 pgs: 64 unknown, 292 active+clean; 583 KiB data, 562 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:06:30.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi072-82404-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:30.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:06:30.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-27"}]: dispatch 2024-03-26T02:06:30.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:30 smithi072 ceph-mon[30177]: osdmap e224: 8 total, 8 up, 8 in 2024-03-26T02:06:30.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-27"}]: dispatch 2024-03-26T02:06:30.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:30 smithi072 ceph-mon[41619]: pgmap v295: 356 pgs: 64 unknown, 292 active+clean; 583 KiB data, 562 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:06:30.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi072-82404-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:30.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:06:30.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-27"}]: dispatch 2024-03-26T02:06:30.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:30 smithi072 ceph-mon[41619]: osdmap e224: 8 total, 8 up, 8 in 2024-03-26T02:06:30.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-27"}]: dispatch 2024-03-26T02:06:31.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:06:30 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:06:31.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:30 smithi173 ceph-mon[33839]: pgmap v295: 356 pgs: 64 unknown, 292 active+clean; 583 KiB data, 562 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:06:31.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi072-82404-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:31.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:06:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-27"}]: dispatch 2024-03-26T02:06:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:30 smithi173 ceph-mon[33839]: osdmap e224: 8 total, 8 up, 8 in 2024-03-26T02:06:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-27"}]: dispatch 2024-03-26T02:06:31.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:31 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:06:31.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:31 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi072-82387-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi072-82387-36"}]': finished 2024-03-26T02:06:31.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:31 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-27"}]': finished 2024-03-26T02:06:31.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:31 smithi072 ceph-mon[30177]: osdmap e225: 8 total, 8 up, 8 in 2024-03-26T02:06:31.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:31 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-27", "mode": "writeback"}]: dispatch 2024-03-26T02:06:31.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:31 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-27", "mode": "writeback"}]: dispatch 2024-03-26T02:06:31.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:31 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:06:31.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:31 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi072-82387-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi072-82387-36"}]': finished 2024-03-26T02:06:31.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:31 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-27"}]': finished 2024-03-26T02:06:31.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:31 smithi072 ceph-mon[41619]: osdmap e225: 8 total, 8 up, 8 in 2024-03-26T02:06:31.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:31 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-27", "mode": "writeback"}]: dispatch 2024-03-26T02:06:31.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:31 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-27", "mode": "writeback"}]: dispatch 2024-03-26T02:06:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:31 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:06:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:31 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi072-82387-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi072-82387-36"}]': finished 2024-03-26T02:06:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:31 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-27"}]': finished 2024-03-26T02:06:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:31 smithi173 ceph-mon[33839]: osdmap e225: 8 total, 8 up, 8 in 2024-03-26T02:06:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:31 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-27", "mode": "writeback"}]: dispatch 2024-03-26T02:06:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:31 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-27", "mode": "writeback"}]: dispatch 2024-03-26T02:06:32.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:32 smithi072 ceph-mon[30177]: pgmap v298: 332 pgs: 8 unknown, 324 active+clean; 584 KiB data, 550 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-26T02:06:32.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:32 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:06:32.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:32 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-27", "mode": "writeback"}]': finished 2024-03-26T02:06:32.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:32 smithi072 ceph-mon[30177]: osdmap e226: 8 total, 8 up, 8 in 2024-03-26T02:06:32.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1087360272' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi072-82404-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:32.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:32 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi072-82404-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:32.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:32 smithi072 ceph-mon[41619]: pgmap v298: 332 pgs: 8 unknown, 324 active+clean; 584 KiB data, 550 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-26T02:06:32.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:32 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:06:32.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:32 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-27", "mode": "writeback"}]': finished 2024-03-26T02:06:32.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:32 smithi072 ceph-mon[41619]: osdmap e226: 8 total, 8 up, 8 in 2024-03-26T02:06:32.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1087360272' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi072-82404-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:32.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:32 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi072-82404-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:33.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:32 smithi173 ceph-mon[33839]: pgmap v298: 332 pgs: 8 unknown, 324 active+clean; 584 KiB data, 550 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-26T02:06:33.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:32 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:06:33.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:32 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-27", "mode": "writeback"}]': finished 2024-03-26T02:06:33.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:32 smithi173 ceph-mon[33839]: osdmap e226: 8 total, 8 up, 8 in 2024-03-26T02:06:33.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1087360272' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi072-82404-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:33.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:32 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi072-82404-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:34.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:33 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi072-82404-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:34.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:33 smithi173 ceph-mon[33839]: osdmap e227: 8 total, 8 up, 8 in 2024-03-26T02:06:34.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:33 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2918319052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:34.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:33 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:33 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:33 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:33 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi072-82387-36"}]': finished 2024-03-26T02:06:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:33 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:06:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:33 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2918319052' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:33 smithi173 ceph-mon[33839]: osdmap e228: 8 total, 8 up, 8 in 2024-03-26T02:06:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:33 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:33 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-27"}]: dispatch 2024-03-26T02:06:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:33 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-27"}]: dispatch 2024-03-26T02:06:34.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:33 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi072-82404-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:34.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:33 smithi072 ceph-mon[30177]: osdmap e227: 8 total, 8 up, 8 in 2024-03-26T02:06:34.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:33 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2918319052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:34.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:33 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:34.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:33 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:34.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:33 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:34.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:33 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi072-82387-36"}]': finished 2024-03-26T02:06:34.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:33 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:06:34.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:33 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2918319052' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:34.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:33 smithi072 ceph-mon[30177]: osdmap e228: 8 total, 8 up, 8 in 2024-03-26T02:06:34.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:33 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:34.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:33 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-27"}]: dispatch 2024-03-26T02:06:34.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:33 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-27"}]: dispatch 2024-03-26T02:06:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:33 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi072-82404-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:33 smithi072 ceph-mon[41619]: osdmap e227: 8 total, 8 up, 8 in 2024-03-26T02:06:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:33 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2918319052' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:33 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:33 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:33 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:33 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi072-82387-36"}]': finished 2024-03-26T02:06:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:33 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:06:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:33 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2918319052' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:33 smithi072 ceph-mon[41619]: osdmap e228: 8 total, 8 up, 8 in 2024-03-26T02:06:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:33 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi072-82387-36"}]: dispatch 2024-03-26T02:06:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:33 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-27"}]: dispatch 2024-03-26T02:06:34.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:33 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-27"}]: dispatch 2024-03-26T02:06:34.983 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:34 smithi072 ceph-mon[30177]: pgmap v301: 356 pgs: 32 unknown, 324 active+clean; 584 KiB data, 550 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T02:06:34.983 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:34 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:06:34.983 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:34 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi072-82404-46"}]: dispatch 2024-03-26T02:06:34.983 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:34 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi072-82404-46"}]: dispatch 2024-03-26T02:06:34.983 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:34 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi072-82404-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:34.983 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:34 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:06:34.984 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:34 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi072-82387-36"}]': finished 2024-03-26T02:06:34.984 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:34 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-27"}]': finished 2024-03-26T02:06:34.984 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:34 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi072-82404-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:34.984 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:34 smithi072 ceph-mon[30177]: osdmap e229: 8 total, 8 up, 8 in 2024-03-26T02:06:34.984 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:34 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi072-82404-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi072-82404-46"}]: dispatch 2024-03-26T02:06:34.984 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:34 smithi072 ceph-mon[41619]: pgmap v301: 356 pgs: 32 unknown, 324 active+clean; 584 KiB data, 550 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T02:06:34.984 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:34 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:06:34.984 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:34 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi072-82404-46"}]: dispatch 2024-03-26T02:06:34.984 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:34 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi072-82404-46"}]: dispatch 2024-03-26T02:06:34.984 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:34 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi072-82404-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:34.984 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:34 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:06:34.985 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:34 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi072-82387-36"}]': finished 2024-03-26T02:06:34.985 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:34 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-27"}]': finished 2024-03-26T02:06:34.985 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:34 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi072-82404-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:34.985 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:34 smithi072 ceph-mon[41619]: osdmap e229: 8 total, 8 up, 8 in 2024-03-26T02:06:34.985 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:34 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi072-82404-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi072-82404-46"}]: dispatch 2024-03-26T02:06:35.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:34 smithi173 ceph-mon[33839]: pgmap v301: 356 pgs: 32 unknown, 324 active+clean; 584 KiB data, 550 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T02:06:35.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:34 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:06:35.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:34 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi072-82404-46"}]: dispatch 2024-03-26T02:06:35.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:34 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi072-82404-46"}]: dispatch 2024-03-26T02:06:35.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:34 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi072-82404-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:35.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:34 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:06:35.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:34 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi072-82387-36"}]': finished 2024-03-26T02:06:35.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:34 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-27"}]': finished 2024-03-26T02:06:35.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:34 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi072-82404-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:35.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:34 smithi173 ceph-mon[33839]: osdmap e229: 8 total, 8 up, 8 in 2024-03-26T02:06:35.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:34 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi072-82404-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi072-82404-46"}]: dispatch 2024-03-26T02:06:35.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:06:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:06:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:06:36.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:36 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3038273670' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:36.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:36 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:36.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:36 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3038273670' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:36.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:36 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:36.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:36 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3038273670' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi072-82387-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:36.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:36 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi072-82387-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:36.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:36 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi072-82387-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:36.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:36 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3038273670' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:36.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:36 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:36.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:36 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3038273670' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:36.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:36 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:36.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:36 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3038273670' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi072-82387-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:36.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:36 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi072-82387-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:36.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:36 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi072-82387-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:36.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:36 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3038273670' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:36.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:36 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:36.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:36 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3038273670' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:36.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:36 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:36.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:36 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3038273670' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi072-82387-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:36.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:36 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi072-82387-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:36.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:36 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi072-82387-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:37.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:37 smithi072 ceph-mon[30177]: pgmap v304: 324 pgs: 324 active+clean; 584 KiB data, 551 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T02:06:37.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:37 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3038273670' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi072-82387-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:37.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:37 smithi072 ceph-mon[30177]: osdmap e230: 8 total, 8 up, 8 in 2024-03-26T02:06:37.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:37 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi072-82387-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:37.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:37 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:06:37.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:37 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi072-82404-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi072-82404-46"}]': finished 2024-03-26T02:06:37.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:37 smithi072 ceph-mon[30177]: osdmap e231: 8 total, 8 up, 8 in 2024-03-26T02:06:37.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:37 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:37.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:37 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:37.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:37 smithi072 ceph-mon[41619]: pgmap v304: 324 pgs: 324 active+clean; 584 KiB data, 551 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T02:06:37.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:37 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3038273670' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi072-82387-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:37.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:37 smithi072 ceph-mon[41619]: osdmap e230: 8 total, 8 up, 8 in 2024-03-26T02:06:37.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:37 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi072-82387-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:37.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:37 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:06:37.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:37 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi072-82404-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi072-82404-46"}]': finished 2024-03-26T02:06:37.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:37 smithi072 ceph-mon[41619]: osdmap e231: 8 total, 8 up, 8 in 2024-03-26T02:06:37.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:37 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:37.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:37 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:37 smithi173 ceph-mon[33839]: pgmap v304: 324 pgs: 324 active+clean; 584 KiB data, 551 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T02:06:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:37 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3038273670' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi072-82387-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:37 smithi173 ceph-mon[33839]: osdmap e230: 8 total, 8 up, 8 in 2024-03-26T02:06:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:37 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi072-82387-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:37.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:37 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:06:37.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:37 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi072-82404-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi072-82404-46"}]': finished 2024-03-26T02:06:37.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:37 smithi173 ceph-mon[33839]: osdmap e231: 8 total, 8 up, 8 in 2024-03-26T02:06:37.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:37 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:37.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:37 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:39.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:38 smithi173 ceph-mon[33839]: pgmap v307: 332 pgs: 40 unknown, 292 active+clean; 584 KiB data, 551 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:06:39.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:38 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi072-82387-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi072-82387-37"}]': finished 2024-03-26T02:06:39.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:38 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:39.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:38 smithi173 ceph-mon[33839]: osdmap e232: 8 total, 8 up, 8 in 2024-03-26T02:06:39.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:38 smithi072 ceph-mon[30177]: pgmap v307: 332 pgs: 40 unknown, 292 active+clean; 584 KiB data, 551 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:06:39.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:38 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi072-82387-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi072-82387-37"}]': finished 2024-03-26T02:06:39.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:38 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:39.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:38 smithi072 ceph-mon[30177]: osdmap e232: 8 total, 8 up, 8 in 2024-03-26T02:06:39.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:38 smithi072 ceph-mon[41619]: pgmap v307: 332 pgs: 40 unknown, 292 active+clean; 584 KiB data, 551 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:06:39.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:38 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi072-82387-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi072-82387-37"}]': finished 2024-03-26T02:06:39.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:38 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:39.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:38 smithi072 ceph-mon[41619]: osdmap e232: 8 total, 8 up, 8 in 2024-03-26T02:06:40.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:39 smithi173 ceph-mon[33839]: osdmap e233: 8 total, 8 up, 8 in 2024-03-26T02:06:40.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi072-82404-46"}]: dispatch 2024-03-26T02:06:40.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:39 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:06:40.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi072-82404-46"}]': finished 2024-03-26T02:06:40.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:39 smithi173 ceph-mon[33839]: osdmap e234: 8 total, 8 up, 8 in 2024-03-26T02:06:40.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi072-82404-46"}]: dispatch 2024-03-26T02:06:40.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3038273670' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:40.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:39 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:40.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:39 smithi072 ceph-mon[30177]: osdmap e233: 8 total, 8 up, 8 in 2024-03-26T02:06:40.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi072-82404-46"}]: dispatch 2024-03-26T02:06:40.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:39 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:06:40.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi072-82404-46"}]': finished 2024-03-26T02:06:40.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:39 smithi072 ceph-mon[30177]: osdmap e234: 8 total, 8 up, 8 in 2024-03-26T02:06:40.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi072-82404-46"}]: dispatch 2024-03-26T02:06:40.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3038273670' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:40.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:39 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:40.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:39 smithi072 ceph-mon[41619]: osdmap e233: 8 total, 8 up, 8 in 2024-03-26T02:06:40.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi072-82404-46"}]: dispatch 2024-03-26T02:06:40.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:39 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:06:40.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi072-82404-46"}]': finished 2024-03-26T02:06:40.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:39 smithi072 ceph-mon[41619]: osdmap e234: 8 total, 8 up, 8 in 2024-03-26T02:06:40.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi072-82404-46"}]: dispatch 2024-03-26T02:06:40.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3038273670' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:40.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:39 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:40.964 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:06:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:06:40.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:40 smithi173 ceph-mon[33839]: pgmap v310: 332 pgs: 40 unknown, 292 active+clean; 584 KiB data, 551 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:06:40.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:40 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:06:40.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:40.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:40 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:40.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:40 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:06:40.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:40 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:06:40.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi072-82404-46"}]': finished 2024-03-26T02:06:40.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:40 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi072-82387-37"}]': finished 2024-03-26T02:06:40.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:40 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:06:40.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:40 smithi173 ceph-mon[33839]: osdmap e235: 8 total, 8 up, 8 in 2024-03-26T02:06:40.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-29"}]: dispatch 2024-03-26T02:06:40.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3038273670' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:40.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:40 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-29"}]: dispatch 2024-03-26T02:06:40.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:40 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:40.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1764842222' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:40.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:40 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:40.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1764842222' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:40.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:40 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:40.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1764842222' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi072-82404-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:40.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:40 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi072-82404-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:41.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[41619]: pgmap v310: 332 pgs: 40 unknown, 292 active+clean; 584 KiB data, 551 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:06:41.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:06:41.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:41.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:06:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:06:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi072-82404-46"}]': finished 2024-03-26T02:06:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi072-82387-37"}]': finished 2024-03-26T02:06:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:06:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[41619]: osdmap e235: 8 total, 8 up, 8 in 2024-03-26T02:06:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-29"}]: dispatch 2024-03-26T02:06:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3038273670' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-29"}]: dispatch 2024-03-26T02:06:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1764842222' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1764842222' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:41.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1764842222' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi072-82404-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:41.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi072-82404-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:41.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[30177]: pgmap v310: 332 pgs: 40 unknown, 292 active+clean; 584 KiB data, 551 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:06:41.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:06:41.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:41.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:41.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:06:41.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:06:41.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/253882414' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi072-82404-46"}]': finished 2024-03-26T02:06:41.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi072-82387-37"}]': finished 2024-03-26T02:06:41.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:06:41.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[30177]: osdmap e235: 8 total, 8 up, 8 in 2024-03-26T02:06:41.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-29"}]: dispatch 2024-03-26T02:06:41.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3038273670' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:41.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-29"}]: dispatch 2024-03-26T02:06:41.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi072-82387-37"}]: dispatch 2024-03-26T02:06:41.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1764842222' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:41.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:41.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1764842222' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:41.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:41.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1764842222' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi072-82404-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:41.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:40 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi072-82404-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:42.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:41 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:06:42.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:41 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:06:42.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:41 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:06:42.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:41 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:06:42.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:41 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-29"}]': finished 2024-03-26T02:06:42.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:41 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi072-82387-37"}]': finished 2024-03-26T02:06:42.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:41 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi072-82404-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:42.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:41 smithi173 ceph-mon[33839]: osdmap e236: 8 total, 8 up, 8 in 2024-03-26T02:06:42.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:41 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1764842222' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi072-82404-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:42.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:41 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-29", "mode": "writeback"}]: dispatch 2024-03-26T02:06:42.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:41 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi072-82404-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:42.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:41 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-29", "mode": "writeback"}]: dispatch 2024-03-26T02:06:42.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:41 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1538509216' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:42.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:41 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:42.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:41 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1538509216' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:42.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:41 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:42.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:41 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1538509216' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi072-82387-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:42.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:41 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi072-82387-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:06:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:06:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:06:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:06:42.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-29"}]': finished 2024-03-26T02:06:42.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi072-82387-37"}]': finished 2024-03-26T02:06:42.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi072-82404-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:42.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[30177]: osdmap e236: 8 total, 8 up, 8 in 2024-03-26T02:06:42.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1764842222' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi072-82404-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:42.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-29", "mode": "writeback"}]: dispatch 2024-03-26T02:06:42.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi072-82404-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:42.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-29", "mode": "writeback"}]: dispatch 2024-03-26T02:06:42.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1538509216' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:42.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:42.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1538509216' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:42.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:42.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1538509216' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi072-82387-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:42.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi072-82387-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:42.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:06:42.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:06:42.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:06:42.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:06:42.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-29"}]': finished 2024-03-26T02:06:42.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi072-82387-37"}]': finished 2024-03-26T02:06:42.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi072-82404-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:42.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[41619]: osdmap e236: 8 total, 8 up, 8 in 2024-03-26T02:06:42.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1764842222' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi072-82404-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:42.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-29", "mode": "writeback"}]: dispatch 2024-03-26T02:06:42.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi072-82404-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:42.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-29", "mode": "writeback"}]: dispatch 2024-03-26T02:06:42.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1538509216' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:42.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:42.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1538509216' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:42.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:42.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1538509216' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi072-82387-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:42.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:41 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi072-82387-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:43.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:42 smithi072 ceph-mon[30177]: pgmap v313: 324 pgs: 324 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-26T02:06:43.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:42 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:06:43.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:42 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-29", "mode": "writeback"}]': finished 2024-03-26T02:06:43.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:42 smithi072 ceph-mon[41619]: pgmap v313: 324 pgs: 324 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-26T02:06:43.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:42 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:06:43.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:42 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-29", "mode": "writeback"}]': finished 2024-03-26T02:06:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:42 smithi173 ceph-mon[33839]: pgmap v313: 324 pgs: 324 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-26T02:06:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:42 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:06:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:42 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-29", "mode": "writeback"}]': finished 2024-03-26T02:06:44.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:43 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi072-82387-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:44.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:43 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1538509216' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi072-82387-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:44.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:43 smithi072 ceph-mon[30177]: osdmap e237: 8 total, 8 up, 8 in 2024-03-26T02:06:44.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:43 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi072-82387-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:44.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:43 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi072-82404-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi072-82404-47"}]': finished 2024-03-26T02:06:44.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:43 smithi072 ceph-mon[30177]: osdmap e238: 8 total, 8 up, 8 in 2024-03-26T02:06:44.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:43 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi072-82387-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:44.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:43 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1538509216' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi072-82387-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:44.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:43 smithi072 ceph-mon[41619]: osdmap e237: 8 total, 8 up, 8 in 2024-03-26T02:06:44.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:43 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi072-82387-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:44.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:43 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi072-82404-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi072-82404-47"}]': finished 2024-03-26T02:06:44.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:43 smithi072 ceph-mon[41619]: osdmap e238: 8 total, 8 up, 8 in 2024-03-26T02:06:44.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:43 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi072-82387-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:44.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:43 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1538509216' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi072-82387-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:44.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:43 smithi173 ceph-mon[33839]: osdmap e237: 8 total, 8 up, 8 in 2024-03-26T02:06:44.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:43 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi072-82387-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:44.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:43 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi072-82404-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi072-82404-47"}]': finished 2024-03-26T02:06:44.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:43 smithi173 ceph-mon[33839]: osdmap e238: 8 total, 8 up, 8 in 2024-03-26T02:06:45.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:44 smithi072 ceph-mon[30177]: pgmap v316: 324 pgs: 324 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-03-26T02:06:45.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:44 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi072-82387-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi072-82387-38"}]': finished 2024-03-26T02:06:45.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:44 smithi072 ceph-mon[30177]: osdmap e239: 8 total, 8 up, 8 in 2024-03-26T02:06:45.220 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:06:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:06:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:06:45.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:44 smithi072 ceph-mon[41619]: pgmap v316: 324 pgs: 324 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-03-26T02:06:45.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:44 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi072-82387-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi072-82387-38"}]': finished 2024-03-26T02:06:45.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:44 smithi072 ceph-mon[41619]: osdmap e239: 8 total, 8 up, 8 in 2024-03-26T02:06:45.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:44 smithi173 ceph-mon[33839]: pgmap v316: 324 pgs: 324 active+clean; 4.6 MiB data, 564 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.0 MiB/s wr, 1 op/s 2024-03-26T02:06:45.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:44 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi072-82387-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi072-82387-38"}]': finished 2024-03-26T02:06:45.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:44 smithi173 ceph-mon[33839]: osdmap e239: 8 total, 8 up, 8 in 2024-03-26T02:06:46.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:45 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:06:46.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:45 smithi072 ceph-mon[30177]: osdmap e240: 8 total, 8 up, 8 in 2024-03-26T02:06:46.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:45 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1764842222' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:46.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:45 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:46.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:45 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:46.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:45 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:45 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:06:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:45 smithi072 ceph-mon[41619]: osdmap e240: 8 total, 8 up, 8 in 2024-03-26T02:06:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:45 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1764842222' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:45 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:45 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:45 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:46.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:45 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:06:46.354 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:45 smithi173 ceph-mon[33839]: osdmap e240: 8 total, 8 up, 8 in 2024-03-26T02:06:46.354 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:45 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1764842222' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:46.354 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:45 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:46.354 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:45 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:46.354 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:45 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:47.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:46 smithi072 ceph-mon[41619]: pgmap v319: 340 pgs: 8 creating+peering, 332 active+clean; 8.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2024-03-26T02:06:47.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:46 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi072-82404-47"}]': finished 2024-03-26T02:06:47.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:46 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:06:47.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:46 smithi072 ceph-mon[41619]: osdmap e241: 8 total, 8 up, 8 in 2024-03-26T02:06:47.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:46 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1764842222' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:47.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:46 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-29"}]: dispatch 2024-03-26T02:06:47.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:46 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:47.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:46 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-29"}]: dispatch 2024-03-26T02:06:47.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:46 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1538509216' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:47.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:46 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:47.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:46 smithi072 ceph-mon[30177]: pgmap v319: 340 pgs: 8 creating+peering, 332 active+clean; 8.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2024-03-26T02:06:47.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:46 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi072-82404-47"}]': finished 2024-03-26T02:06:47.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:46 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:06:47.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:46 smithi072 ceph-mon[30177]: osdmap e241: 8 total, 8 up, 8 in 2024-03-26T02:06:47.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:46 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1764842222' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:47.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:46 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-29"}]: dispatch 2024-03-26T02:06:47.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:46 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:47.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:46 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-29"}]: dispatch 2024-03-26T02:06:47.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:46 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1538509216' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:47.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:46 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:47.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:46 smithi173 ceph-mon[33839]: pgmap v319: 340 pgs: 8 creating+peering, 332 active+clean; 8.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 255 B/s wr, 26 op/s 2024-03-26T02:06:47.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:46 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi072-82404-47"}]': finished 2024-03-26T02:06:47.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:46 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:06:47.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:46 smithi173 ceph-mon[33839]: osdmap e241: 8 total, 8 up, 8 in 2024-03-26T02:06:47.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:46 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1764842222' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:47.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:46 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-29"}]: dispatch 2024-03-26T02:06:47.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:46 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi072-82404-47"}]: dispatch 2024-03-26T02:06:47.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:46 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-29"}]: dispatch 2024-03-26T02:06:47.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:46 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1538509216' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:47.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:46 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:48.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:47 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:06:48.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:47 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi072-82404-47"}]': finished 2024-03-26T02:06:48.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:47 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-29"}]': finished 2024-03-26T02:06:48.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:47 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi072-82387-38"}]': finished 2024-03-26T02:06:48.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:47 smithi072 ceph-mon[41619]: osdmap e242: 8 total, 8 up, 8 in 2024-03-26T02:06:48.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:47 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1538509216' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:48.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:47 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:48.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:47 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3364328915' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:48.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:47 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:48.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:47 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3364328915' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:48.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:47 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:48.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:47 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3364328915' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi072-82404-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:48.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:47 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi072-82404-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:48.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:47 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:06:48.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:47 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi072-82404-47"}]': finished 2024-03-26T02:06:48.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:47 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-29"}]': finished 2024-03-26T02:06:48.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:47 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi072-82387-38"}]': finished 2024-03-26T02:06:48.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:47 smithi072 ceph-mon[30177]: osdmap e242: 8 total, 8 up, 8 in 2024-03-26T02:06:48.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:47 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1538509216' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:48.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:47 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:48.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:47 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3364328915' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:48.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:47 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:48.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:47 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3364328915' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:48.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:47 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:48.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:47 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3364328915' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi072-82404-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:48.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:47 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi072-82404-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:48.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:47 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:06:48.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:47 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi072-82404-47"}]': finished 2024-03-26T02:06:48.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:47 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-29"}]': finished 2024-03-26T02:06:48.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:47 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi072-82387-38"}]': finished 2024-03-26T02:06:48.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:47 smithi173 ceph-mon[33839]: osdmap e242: 8 total, 8 up, 8 in 2024-03-26T02:06:48.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:47 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1538509216' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:48.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:47 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi072-82387-38"}]: dispatch 2024-03-26T02:06:48.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:47 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3364328915' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:48.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:47 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:48.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:47 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3364328915' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:48.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:47 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:48.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:47 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3364328915' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi072-82404-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:48.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:47 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi072-82404-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:49.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:48 smithi072 ceph-mon[30177]: pgmap v322: 324 pgs: 324 active+clean; 8.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 26 op/s 2024-03-26T02:06:49.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:48 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi072-82387-38"}]': finished 2024-03-26T02:06:49.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:48 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi072-82404-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:49.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:48 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3364328915' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi072-82404-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:49.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:48 smithi072 ceph-mon[30177]: osdmap e243: 8 total, 8 up, 8 in 2024-03-26T02:06:49.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:48 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi072-82404-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:49.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:48 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1750270755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:49.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:48 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:49.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:48 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1750270755' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:49.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:48 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:49.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:48 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1750270755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi072-82387-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:49.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:48 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi072-82387-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:49.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:48 smithi072 ceph-mon[41619]: pgmap v322: 324 pgs: 324 active+clean; 8.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 26 op/s 2024-03-26T02:06:49.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:48 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi072-82387-38"}]': finished 2024-03-26T02:06:49.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:48 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi072-82404-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:49.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:48 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3364328915' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi072-82404-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:49.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:48 smithi072 ceph-mon[41619]: osdmap e243: 8 total, 8 up, 8 in 2024-03-26T02:06:49.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:48 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi072-82404-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:49.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:48 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1750270755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:49.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:48 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:49.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:48 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1750270755' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:49.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:48 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:49.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:48 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1750270755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi072-82387-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:49.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:48 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi072-82387-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:48 smithi173 ceph-mon[33839]: pgmap v322: 324 pgs: 324 active+clean; 8.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 0 B/s wr, 26 op/s 2024-03-26T02:06:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:48 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi072-82387-38"}]': finished 2024-03-26T02:06:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:48 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi072-82404-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:48 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3364328915' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi072-82404-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:48 smithi173 ceph-mon[33839]: osdmap e243: 8 total, 8 up, 8 in 2024-03-26T02:06:49.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:48 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi072-82404-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:49.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:48 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1750270755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:49.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:48 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:49.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:48 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1750270755' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:49.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:48 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:49.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:48 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1750270755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi072-82387-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:49.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:48 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi072-82387-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:51.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:06:50 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:06:51.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:50 smithi173 ceph-mon[33839]: pgmap v325: 292 pgs: 292 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:06:51.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:50 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi072-82387-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:51.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:50 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1750270755' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi072-82387-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:51.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:50 smithi173 ceph-mon[33839]: osdmap e244: 8 total, 8 up, 8 in 2024-03-26T02:06:51.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:50 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:51.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:50 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi072-82387-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:51.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:50 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:51.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:50 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi072-82404-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi072-82404-48"}]': finished 2024-03-26T02:06:51.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:50 smithi072 ceph-mon[30177]: pgmap v325: 292 pgs: 292 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:06:51.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:50 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi072-82387-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:51.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:50 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1750270755' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi072-82387-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:51.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:50 smithi072 ceph-mon[30177]: osdmap e244: 8 total, 8 up, 8 in 2024-03-26T02:06:51.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:50 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:51.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:50 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi072-82387-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:51.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:50 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:51.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:50 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi072-82404-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi072-82404-48"}]': finished 2024-03-26T02:06:51.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:50 smithi072 ceph-mon[41619]: pgmap v325: 292 pgs: 292 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:06:51.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:50 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi072-82387-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:51.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:50 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1750270755' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi072-82387-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:51.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:50 smithi072 ceph-mon[41619]: osdmap e244: 8 total, 8 up, 8 in 2024-03-26T02:06:51.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:50 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:51.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:50 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi072-82387-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:51.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:50 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:51.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:50 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi072-82404-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi072-82404-48"}]': finished 2024-03-26T02:06:52.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:51 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:52.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:51 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:52.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:51 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:06:52.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:51 smithi072 ceph-mon[30177]: osdmap e245: 8 total, 8 up, 8 in 2024-03-26T02:06:52.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:51 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:52.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:51 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:06:52.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:51 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi072-82387-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi072-82387-39"}]': finished 2024-03-26T02:06:52.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:51 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:06:52.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:51 smithi072 ceph-mon[30177]: osdmap e246: 8 total, 8 up, 8 in 2024-03-26T02:06:52.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:51 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-31"}]: dispatch 2024-03-26T02:06:52.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:51 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-31"}]: dispatch 2024-03-26T02:06:52.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:51 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:52.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:51 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:52.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:51 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:06:52.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:51 smithi072 ceph-mon[41619]: osdmap e245: 8 total, 8 up, 8 in 2024-03-26T02:06:52.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:51 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:52.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:51 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:06:52.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:51 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi072-82387-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi072-82387-39"}]': finished 2024-03-26T02:06:52.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:51 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:06:52.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:51 smithi072 ceph-mon[41619]: osdmap e246: 8 total, 8 up, 8 in 2024-03-26T02:06:52.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:51 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-31"}]: dispatch 2024-03-26T02:06:52.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:51 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-31"}]: dispatch 2024-03-26T02:06:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:51 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:51 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:51 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:06:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:51 smithi173 ceph-mon[33839]: osdmap e245: 8 total, 8 up, 8 in 2024-03-26T02:06:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:51 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:51 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:06:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:51 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi072-82387-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi072-82387-39"}]': finished 2024-03-26T02:06:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:51 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:06:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:51 smithi173 ceph-mon[33839]: osdmap e246: 8 total, 8 up, 8 in 2024-03-26T02:06:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:51 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-31"}]: dispatch 2024-03-26T02:06:52.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:51 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-31"}]: dispatch 2024-03-26T02:06:53.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:52 smithi072 ceph-mon[41619]: pgmap v328: 332 pgs: 8 unknown, 32 creating+peering, 292 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:06:53.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:52 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:06:53.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:52 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-31"}]': finished 2024-03-26T02:06:53.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:52 smithi072 ceph-mon[41619]: osdmap e247: 8 total, 8 up, 8 in 2024-03-26T02:06:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:52 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-31", "mode": "writeback"}]: dispatch 2024-03-26T02:06:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:52 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-31", "mode": "writeback"}]: dispatch 2024-03-26T02:06:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:52 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3364328915' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:52 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:53.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:52 smithi072 ceph-mon[30177]: pgmap v328: 332 pgs: 8 unknown, 32 creating+peering, 292 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:06:53.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:52 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:06:53.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:52 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-31"}]': finished 2024-03-26T02:06:53.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:52 smithi072 ceph-mon[30177]: osdmap e247: 8 total, 8 up, 8 in 2024-03-26T02:06:53.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:52 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-31", "mode": "writeback"}]: dispatch 2024-03-26T02:06:53.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:52 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-31", "mode": "writeback"}]: dispatch 2024-03-26T02:06:53.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:52 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3364328915' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:53.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:52 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:52 smithi173 ceph-mon[33839]: pgmap v328: 332 pgs: 8 unknown, 32 creating+peering, 292 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:06:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:52 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:06:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:52 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-31"}]': finished 2024-03-26T02:06:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:52 smithi173 ceph-mon[33839]: osdmap e247: 8 total, 8 up, 8 in 2024-03-26T02:06:53.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:52 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-31", "mode": "writeback"}]: dispatch 2024-03-26T02:06:53.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:52 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-31", "mode": "writeback"}]: dispatch 2024-03-26T02:06:53.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:52 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3364328915' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:53.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:52 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:54.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:53 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:06:54.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:53 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-31", "mode": "writeback"}]': finished 2024-03-26T02:06:54.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:53 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi072-82404-48"}]': finished 2024-03-26T02:06:54.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:53 smithi072 ceph-mon[41619]: osdmap e248: 8 total, 8 up, 8 in 2024-03-26T02:06:54.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:53 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3364328915' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:54.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:53 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1750270755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:54.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:53 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:54.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:53 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:54.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:53 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:54.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:53 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:54.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:53 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:06:54.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:53 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-31", "mode": "writeback"}]': finished 2024-03-26T02:06:54.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:53 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi072-82404-48"}]': finished 2024-03-26T02:06:54.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:53 smithi072 ceph-mon[30177]: osdmap e248: 8 total, 8 up, 8 in 2024-03-26T02:06:54.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:53 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3364328915' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:54.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:53 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1750270755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:54.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:53 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:54.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:53 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:54.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:53 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:54.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:53 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:54.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:53 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:06:54.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:53 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-31", "mode": "writeback"}]': finished 2024-03-26T02:06:54.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:53 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi072-82404-48"}]': finished 2024-03-26T02:06:54.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:53 smithi173 ceph-mon[33839]: osdmap e248: 8 total, 8 up, 8 in 2024-03-26T02:06:54.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:53 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3364328915' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:54.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:53 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1750270755' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:54.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:53 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:54.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:53 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi072-82404-48"}]: dispatch 2024-03-26T02:06:54.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:53 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:54.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:53 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:06:55.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:06:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:06:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:06:55.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:54 smithi072 ceph-mon[30177]: pgmap v331: 332 pgs: 8 unknown, 32 creating+peering, 292 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-26T02:06:55.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:54 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi072-82387-39"}]': finished 2024-03-26T02:06:55.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:54 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi072-82404-48"}]': finished 2024-03-26T02:06:55.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:54 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:06:55.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:54 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1750270755' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:55.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:54 smithi072 ceph-mon[30177]: osdmap e249: 8 total, 8 up, 8 in 2024-03-26T02:06:55.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:54 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-31"}]: dispatch 2024-03-26T02:06:55.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:54 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-31"}]: dispatch 2024-03-26T02:06:55.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:54 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:55.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:54 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi072-82404-49"}]: dispatch 2024-03-26T02:06:55.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:54 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi072-82404-49"}]: dispatch 2024-03-26T02:06:55.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:54 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi072-82404-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:55.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:54 smithi072 ceph-mon[41619]: pgmap v331: 332 pgs: 8 unknown, 32 creating+peering, 292 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-26T02:06:55.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:54 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi072-82387-39"}]': finished 2024-03-26T02:06:55.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:54 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi072-82404-48"}]': finished 2024-03-26T02:06:55.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:54 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:06:55.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:54 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1750270755' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:55.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:54 smithi072 ceph-mon[41619]: osdmap e249: 8 total, 8 up, 8 in 2024-03-26T02:06:55.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:54 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-31"}]: dispatch 2024-03-26T02:06:55.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:54 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-31"}]: dispatch 2024-03-26T02:06:55.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:54 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:55.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:54 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi072-82404-49"}]: dispatch 2024-03-26T02:06:55.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:54 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi072-82404-49"}]: dispatch 2024-03-26T02:06:55.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:54 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi072-82404-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:54 smithi173 ceph-mon[33839]: pgmap v331: 332 pgs: 8 unknown, 32 creating+peering, 292 active+clean; 4.6 MiB data, 565 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-26T02:06:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:54 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi072-82387-39"}]': finished 2024-03-26T02:06:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:54 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi072-82404-48"}]': finished 2024-03-26T02:06:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:54 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:06:55.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:54 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1750270755' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:55.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:54 smithi173 ceph-mon[33839]: osdmap e249: 8 total, 8 up, 8 in 2024-03-26T02:06:55.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:54 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-31"}]: dispatch 2024-03-26T02:06:55.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:54 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-31"}]: dispatch 2024-03-26T02:06:55.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:54 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi072-82387-39"}]: dispatch 2024-03-26T02:06:55.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:54 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi072-82404-49"}]: dispatch 2024-03-26T02:06:55.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:54 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi072-82404-49"}]: dispatch 2024-03-26T02:06:55.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:54 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi072-82404-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:56.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:55 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:06:56.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:55 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-31"}]': finished 2024-03-26T02:06:56.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:55 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi072-82387-39"}]': finished 2024-03-26T02:06:56.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:55 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi072-82404-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:56.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:55 smithi072 ceph-mon[30177]: osdmap e250: 8 total, 8 up, 8 in 2024-03-26T02:06:56.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:55 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi072-82404-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi072-82404-49"}]: dispatch 2024-03-26T02:06:56.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:55 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2532013164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:06:56.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:55 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:06:56.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:55 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2532013164' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:06:56.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:55 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:06:56.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:55 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2532013164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi072-82387-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:56.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:55 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi072-82387-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:56.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:55 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:06:56.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:55 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-31"}]': finished 2024-03-26T02:06:56.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:55 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi072-82387-39"}]': finished 2024-03-26T02:06:56.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:55 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi072-82404-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:56.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:55 smithi072 ceph-mon[41619]: osdmap e250: 8 total, 8 up, 8 in 2024-03-26T02:06:56.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:55 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi072-82404-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi072-82404-49"}]: dispatch 2024-03-26T02:06:56.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:55 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2532013164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:06:56.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:55 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:06:56.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:55 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2532013164' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:06:56.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:55 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:06:56.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:55 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2532013164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi072-82387-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:56.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:55 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi072-82387-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:55 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:06:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:55 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-31"}]': finished 2024-03-26T02:06:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:55 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi072-82387-39"}]': finished 2024-03-26T02:06:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:55 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi072-82404-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:55 smithi173 ceph-mon[33839]: osdmap e250: 8 total, 8 up, 8 in 2024-03-26T02:06:56.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:55 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi072-82404-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi072-82404-49"}]: dispatch 2024-03-26T02:06:56.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:55 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2532013164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:06:56.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:55 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:06:56.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:55 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2532013164' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:06:56.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:55 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:06:56.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:55 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2532013164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi072-82387-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:56.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:55 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi072-82387-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:06:57.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:57 smithi072 ceph-mon[41619]: pgmap v334: 324 pgs: 324 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T02:06:57.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:57 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi072-82387-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:57.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:57 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2532013164' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi072-82387-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:06:57.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:57 smithi072 ceph-mon[41619]: osdmap e251: 8 total, 8 up, 8 in 2024-03-26T02:06:57.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:57 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi072-82387-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:06:57.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:57 smithi072 ceph-mon[30177]: pgmap v334: 324 pgs: 324 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T02:06:57.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:57 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi072-82387-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:57.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:57 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2532013164' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi072-82387-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:06:57.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:57 smithi072 ceph-mon[30177]: osdmap e251: 8 total, 8 up, 8 in 2024-03-26T02:06:57.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:57 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi072-82387-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:06:57.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:57 smithi173 ceph-mon[33839]: pgmap v334: 324 pgs: 324 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T02:06:57.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:57 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi072-82387-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:06:57.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:57 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2532013164' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi072-82387-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:06:57.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:57 smithi173 ceph-mon[33839]: osdmap e251: 8 total, 8 up, 8 in 2024-03-26T02:06:57.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:57 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi072-82387-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:06:58.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:58 smithi072 ceph-mon[41619]: pgmap v337: 292 pgs: 292 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:06:58.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi072-82404-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi072-82404-49"}]': finished 2024-03-26T02:06:58.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:58 smithi072 ceph-mon[41619]: osdmap e252: 8 total, 8 up, 8 in 2024-03-26T02:06:58.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:58.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:58 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:58.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:58 smithi072 ceph-mon[30177]: pgmap v337: 292 pgs: 292 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:06:58.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi072-82404-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi072-82404-49"}]': finished 2024-03-26T02:06:58.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:58 smithi072 ceph-mon[30177]: osdmap e252: 8 total, 8 up, 8 in 2024-03-26T02:06:58.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:58.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:58 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:59.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:58 smithi173 ceph-mon[33839]: pgmap v337: 292 pgs: 292 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:06:59.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi072-82404-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi072-82404-49"}]': finished 2024-03-26T02:06:59.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:58 smithi173 ceph-mon[33839]: osdmap e252: 8 total, 8 up, 8 in 2024-03-26T02:06:59.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:59.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:58 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:06:59.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:59 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi072-82387-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi072-82387-40"}]': finished 2024-03-26T02:06:59.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:59 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:59.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:59 smithi072 ceph-mon[41619]: osdmap e253: 8 total, 8 up, 8 in 2024-03-26T02:06:59.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:59 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:59.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:06:59 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:59.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:59 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi072-82387-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi072-82387-40"}]': finished 2024-03-26T02:06:59.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:59 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:06:59.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:59 smithi072 ceph-mon[30177]: osdmap e253: 8 total, 8 up, 8 in 2024-03-26T02:06:59.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:59 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:06:59.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:06:59 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:00.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:59 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi072-82387-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi072-82387-40"}]': finished 2024-03-26T02:07:00.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:59 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:07:00.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:59 smithi173 ceph-mon[33839]: osdmap e253: 8 total, 8 up, 8 in 2024-03-26T02:07:00.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:59 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:00.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:06:59 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:00.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:00 smithi072 ceph-mon[41619]: pgmap v340: 340 pgs: 48 unknown, 292 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:07:00.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:00 smithi072 ceph-mon[41619]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:00.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:00 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:07:00.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:00 smithi072 ceph-mon[41619]: osdmap e254: 8 total, 8 up, 8 in 2024-03-26T02:07:00.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-33"}]: dispatch 2024-03-26T02:07:00.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:00 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-33"}]: dispatch 2024-03-26T02:07:00.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi072-82404-49"}]: dispatch 2024-03-26T02:07:00.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:00 smithi072 ceph-mon[30177]: pgmap v340: 340 pgs: 48 unknown, 292 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:07:00.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:00 smithi072 ceph-mon[30177]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:00.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:00 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:07:00.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:00 smithi072 ceph-mon[30177]: osdmap e254: 8 total, 8 up, 8 in 2024-03-26T02:07:00.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-33"}]: dispatch 2024-03-26T02:07:00.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:00 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-33"}]: dispatch 2024-03-26T02:07:00.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi072-82404-49"}]: dispatch 2024-03-26T02:07:01.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:07:00 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:07:01.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:00 smithi173 ceph-mon[33839]: pgmap v340: 340 pgs: 48 unknown, 292 active+clean; 4.6 MiB data, 570 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:07:01.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:00 smithi173 ceph-mon[33839]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:01.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:00 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:07:01.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:00 smithi173 ceph-mon[33839]: osdmap e254: 8 total, 8 up, 8 in 2024-03-26T02:07:01.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-33"}]: dispatch 2024-03-26T02:07:01.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:00 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-33"}]: dispatch 2024-03-26T02:07:01.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi072-82404-49"}]: dispatch 2024-03-26T02:07:01.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:01 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-33"}]': finished 2024-03-26T02:07:01.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi072-82404-49"}]': finished 2024-03-26T02:07:01.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:01 smithi072 ceph-mon[30177]: osdmap e255: 8 total, 8 up, 8 in 2024-03-26T02:07:01.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-33", "mode": "writeback"}]: dispatch 2024-03-26T02:07:01.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi072-82404-49"}]: dispatch 2024-03-26T02:07:01.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2532013164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:07:01.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:01 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-33", "mode": "writeback"}]: dispatch 2024-03-26T02:07:01.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:01 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:07:01.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:01 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:07:01.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:01 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-33"}]': finished 2024-03-26T02:07:01.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi072-82404-49"}]': finished 2024-03-26T02:07:01.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:01 smithi072 ceph-mon[41619]: osdmap e255: 8 total, 8 up, 8 in 2024-03-26T02:07:01.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-33", "mode": "writeback"}]: dispatch 2024-03-26T02:07:01.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi072-82404-49"}]: dispatch 2024-03-26T02:07:01.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2532013164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:07:01.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:01 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-33", "mode": "writeback"}]: dispatch 2024-03-26T02:07:01.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:01 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:07:01.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:01 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:07:02.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:01 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-33"}]': finished 2024-03-26T02:07:02.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi072-82404-49"}]': finished 2024-03-26T02:07:02.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:01 smithi173 ceph-mon[33839]: osdmap e255: 8 total, 8 up, 8 in 2024-03-26T02:07:02.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-33", "mode": "writeback"}]: dispatch 2024-03-26T02:07:02.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi072-82404-49"}]: dispatch 2024-03-26T02:07:02.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2532013164' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:07:02.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:01 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-33", "mode": "writeback"}]: dispatch 2024-03-26T02:07:02.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:01 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:07:02.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:01 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:07:02.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[30177]: pgmap v343: 324 pgs: 324 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:02.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:07:02.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi072-82404-49"}]': finished 2024-03-26T02:07:02.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-33", "mode": "writeback"}]': finished 2024-03-26T02:07:02.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi072-82387-40"}]': finished 2024-03-26T02:07:02.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[30177]: osdmap e256: 8 total, 8 up, 8 in 2024-03-26T02:07:02.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2532013164' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:07:02.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:07:02.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3916017095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:02.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:02.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3916017095' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:02.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:02.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3916017095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi072-82404-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:02.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi072-82404-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:02.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:02.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:02.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[41619]: pgmap v343: 324 pgs: 324 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:02.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:07:02.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi072-82404-49"}]': finished 2024-03-26T02:07:02.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-33", "mode": "writeback"}]': finished 2024-03-26T02:07:02.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi072-82387-40"}]': finished 2024-03-26T02:07:02.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[41619]: osdmap e256: 8 total, 8 up, 8 in 2024-03-26T02:07:02.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2532013164' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:07:02.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:07:02.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3916017095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:02.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:02.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3916017095' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:02.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:02.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3916017095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi072-82404-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:02.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi072-82404-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:02.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:02.972 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:02 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:03.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:02 smithi173 ceph-mon[33839]: pgmap v343: 324 pgs: 324 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:03.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:02 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:07:03.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1938009431' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi072-82404-49"}]': finished 2024-03-26T02:07:03.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:02 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-33", "mode": "writeback"}]': finished 2024-03-26T02:07:03.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:02 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi072-82387-40"}]': finished 2024-03-26T02:07:03.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:02 smithi173 ceph-mon[33839]: osdmap e256: 8 total, 8 up, 8 in 2024-03-26T02:07:03.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2532013164' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:07:03.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:02 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi072-82387-40"}]: dispatch 2024-03-26T02:07:03.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3916017095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:03.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:02 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:03.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3916017095' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:03.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:02 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:03.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3916017095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi072-82404-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:03.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:02 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi072-82404-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:03.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:03.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:02 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:03.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:03 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi072-82387-40"}]': finished 2024-03-26T02:07:03.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:03 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi072-82404-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:03.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:03 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:07:03.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:03 smithi072 ceph-mon[30177]: osdmap e257: 8 total, 8 up, 8 in 2024-03-26T02:07:03.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3916017095' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi072-82404-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:03.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-33"}]: dispatch 2024-03-26T02:07:03.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:03 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi072-82404-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:03.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:03 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-33"}]: dispatch 2024-03-26T02:07:03.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi072-82387-41"}]: dispatch 2024-03-26T02:07:03.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi072-82387-41"}]: dispatch 2024-03-26T02:07:03.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi072-82387-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:03.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:03 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi072-82387-40"}]': finished 2024-03-26T02:07:03.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:03 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi072-82404-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:03.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:03 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:07:03.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:03 smithi072 ceph-mon[41619]: osdmap e257: 8 total, 8 up, 8 in 2024-03-26T02:07:03.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3916017095' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi072-82404-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:03.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-33"}]: dispatch 2024-03-26T02:07:03.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:03 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi072-82404-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:03.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:03 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-33"}]: dispatch 2024-03-26T02:07:03.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi072-82387-41"}]: dispatch 2024-03-26T02:07:03.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi072-82387-41"}]: dispatch 2024-03-26T02:07:03.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi072-82387-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:04.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:03 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi072-82387-40"}]': finished 2024-03-26T02:07:04.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:03 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi072-82404-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:04.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:03 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:07:04.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:03 smithi173 ceph-mon[33839]: osdmap e257: 8 total, 8 up, 8 in 2024-03-26T02:07:04.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3916017095' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi072-82404-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:04.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-33"}]: dispatch 2024-03-26T02:07:04.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:03 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi072-82404-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:04.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:03 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-33"}]: dispatch 2024-03-26T02:07:04.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi072-82387-41"}]: dispatch 2024-03-26T02:07:04.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi072-82387-41"}]: dispatch 2024-03-26T02:07:04.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi072-82387-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:04.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:04 smithi072 ceph-mon[30177]: pgmap v346: 324 pgs: 324 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:04.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:04 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:07:04.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:04 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-33"}]': finished 2024-03-26T02:07:04.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:04 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi072-82387-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:04.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:04 smithi072 ceph-mon[30177]: osdmap e258: 8 total, 8 up, 8 in 2024-03-26T02:07:04.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:04 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi072-82387-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi072-82387-41"}]: dispatch 2024-03-26T02:07:04.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:04 smithi072 ceph-mon[41619]: pgmap v346: 324 pgs: 324 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:04.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:04 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:07:04.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:04 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-33"}]': finished 2024-03-26T02:07:04.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:04 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi072-82387-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:04.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:04 smithi072 ceph-mon[41619]: osdmap e258: 8 total, 8 up, 8 in 2024-03-26T02:07:04.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:04 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi072-82387-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi072-82387-41"}]: dispatch 2024-03-26T02:07:05.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:04 smithi173 ceph-mon[33839]: pgmap v346: 324 pgs: 324 active+clean; 4.6 MiB data, 574 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:05.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:04 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:07:05.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:04 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-33"}]': finished 2024-03-26T02:07:05.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:04 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi072-82387-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:05.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:04 smithi173 ceph-mon[33839]: osdmap e258: 8 total, 8 up, 8 in 2024-03-26T02:07:05.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:04 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi072-82387-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi072-82387-41"}]: dispatch 2024-03-26T02:07:05.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:07:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:07:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:07:05.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi072-82404-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi072-82404-50"}]': finished 2024-03-26T02:07:05.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:05 smithi072 ceph-mon[30177]: osdmap e259: 8 total, 8 up, 8 in 2024-03-26T02:07:05.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:05 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:05.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi072-82404-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi072-82404-50"}]': finished 2024-03-26T02:07:05.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:05 smithi072 ceph-mon[41619]: osdmap e259: 8 total, 8 up, 8 in 2024-03-26T02:07:05.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:05 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:06.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi072-82404-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi072-82404-50"}]': finished 2024-03-26T02:07:06.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:05 smithi173 ceph-mon[33839]: osdmap e259: 8 total, 8 up, 8 in 2024-03-26T02:07:06.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:05 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:06.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:06 smithi072 ceph-mon[30177]: pgmap v349: 300 pgs: 8 creating+peering, 292 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:07:06.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi072-82387-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi072-82387-41"}]': finished 2024-03-26T02:07:06.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:06 smithi072 ceph-mon[30177]: osdmap e260: 8 total, 8 up, 8 in 2024-03-26T02:07:06.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:06.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:06 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:06.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:06 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:07:06.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:06 smithi072 ceph-mon[41619]: pgmap v349: 300 pgs: 8 creating+peering, 292 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:07:06.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi072-82387-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi072-82387-41"}]': finished 2024-03-26T02:07:06.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:06 smithi072 ceph-mon[41619]: osdmap e260: 8 total, 8 up, 8 in 2024-03-26T02:07:06.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:06.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:06 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:06.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:06 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:07:07.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:06 smithi173 ceph-mon[33839]: pgmap v349: 300 pgs: 8 creating+peering, 292 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:07:07.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi072-82387-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi072-82387-41"}]': finished 2024-03-26T02:07:07.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:06 smithi173 ceph-mon[33839]: osdmap e260: 8 total, 8 up, 8 in 2024-03-26T02:07:07.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:07.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:06 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:07.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:06 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:07:07.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:07 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:07:07.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:07 smithi072 ceph-mon[41619]: osdmap e261: 8 total, 8 up, 8 in 2024-03-26T02:07:07.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3916017095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:07.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:07 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:07.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:07.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:07 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:07.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:07 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:07:07.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:07 smithi072 ceph-mon[30177]: osdmap e261: 8 total, 8 up, 8 in 2024-03-26T02:07:07.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3916017095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:07.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:07 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:07.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:07.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:07 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:08.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:07 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:07:08.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:07 smithi173 ceph-mon[33839]: osdmap e261: 8 total, 8 up, 8 in 2024-03-26T02:07:08.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3916017095' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:08.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:07 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:08.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:08.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:07 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:09.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:08 smithi173 ceph-mon[33839]: pgmap v352: 332 pgs: 40 unknown, 292 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:07:09.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi072-82404-50"}]': finished 2024-03-26T02:07:09.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:07:09.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:08 smithi173 ceph-mon[33839]: osdmap e262: 8 total, 8 up, 8 in 2024-03-26T02:07:09.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3916017095' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:09.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-35"}]: dispatch 2024-03-26T02:07:09.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi072-82387-41"}]: dispatch 2024-03-26T02:07:09.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:09.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-35"}]: dispatch 2024-03-26T02:07:09.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:08 smithi072 ceph-mon[41619]: pgmap v352: 332 pgs: 40 unknown, 292 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:07:09.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi072-82404-50"}]': finished 2024-03-26T02:07:09.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:07:09.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:08 smithi072 ceph-mon[41619]: osdmap e262: 8 total, 8 up, 8 in 2024-03-26T02:07:09.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3916017095' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:09.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-35"}]: dispatch 2024-03-26T02:07:09.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi072-82387-41"}]: dispatch 2024-03-26T02:07:09.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:09.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-35"}]: dispatch 2024-03-26T02:07:09.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:08 smithi072 ceph-mon[30177]: pgmap v352: 332 pgs: 40 unknown, 292 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:07:09.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi072-82404-50"}]': finished 2024-03-26T02:07:09.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:07:09.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:08 smithi072 ceph-mon[30177]: osdmap e262: 8 total, 8 up, 8 in 2024-03-26T02:07:09.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3916017095' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:09.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-35"}]: dispatch 2024-03-26T02:07:09.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi072-82387-41"}]: dispatch 2024-03-26T02:07:09.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi072-82404-50"}]: dispatch 2024-03-26T02:07:09.224 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-35"}]: dispatch 2024-03-26T02:07:10.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi072-82387-41"}]': finished 2024-03-26T02:07:10.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi072-82404-50"}]': finished 2024-03-26T02:07:10.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-35"}]': finished 2024-03-26T02:07:10.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:09 smithi173 ceph-mon[33839]: osdmap e263: 8 total, 8 up, 8 in 2024-03-26T02:07:10.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi072-82387-41"}]: dispatch 2024-03-26T02:07:10.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-35", "mode": "writeback"}]: dispatch 2024-03-26T02:07:10.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-35", "mode": "writeback"}]: dispatch 2024-03-26T02:07:10.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:09 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:07:10.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi072-82387-41"}]': finished 2024-03-26T02:07:10.120 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi072-82387-41"}]': finished 2024-03-26T02:07:10.121 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi072-82404-50"}]': finished 2024-03-26T02:07:10.121 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-35"}]': finished 2024-03-26T02:07:10.121 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:09 smithi072 ceph-mon[30177]: osdmap e263: 8 total, 8 up, 8 in 2024-03-26T02:07:10.121 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi072-82387-41"}]: dispatch 2024-03-26T02:07:10.121 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-35", "mode": "writeback"}]: dispatch 2024-03-26T02:07:10.121 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-35", "mode": "writeback"}]: dispatch 2024-03-26T02:07:10.121 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:09 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:07:10.121 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi072-82387-41"}]': finished 2024-03-26T02:07:10.121 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi072-82387-41"}]': finished 2024-03-26T02:07:10.121 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi072-82404-50"}]': finished 2024-03-26T02:07:10.122 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-35"}]': finished 2024-03-26T02:07:10.122 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:09 smithi072 ceph-mon[41619]: osdmap e263: 8 total, 8 up, 8 in 2024-03-26T02:07:10.122 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi072-82387-41"}]: dispatch 2024-03-26T02:07:10.122 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-35", "mode": "writeback"}]: dispatch 2024-03-26T02:07:10.122 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-35", "mode": "writeback"}]: dispatch 2024-03-26T02:07:10.122 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:09 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:07:10.122 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3231052419' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi072-82387-41"}]': finished 2024-03-26T02:07:10.965 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:07:10 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:07:10.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:10 smithi173 ceph-mon[33839]: pgmap v355: 324 pgs: 32 unknown, 292 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:07:10.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-35", "mode": "writeback"}]': finished 2024-03-26T02:07:10.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:10 smithi173 ceph-mon[33839]: osdmap e264: 8 total, 8 up, 8 in 2024-03-26T02:07:10.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2927718344' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi072-82404-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:10.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi072-82404-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:10.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi072-82387-42"}]: dispatch 2024-03-26T02:07:10.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi072-82387-42"}]: dispatch 2024-03-26T02:07:10.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi072-82387-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:10.965 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi072-82404-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:07:10.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi072-82387-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:10.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:10 smithi173 ceph-mon[33839]: osdmap e265: 8 total, 8 up, 8 in 2024-03-26T02:07:10.966 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi072-82387-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi072-82387-42"}]: dispatch 2024-03-26T02:07:11.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:10 smithi072 ceph-mon[41619]: pgmap v355: 324 pgs: 32 unknown, 292 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:07:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-35", "mode": "writeback"}]': finished 2024-03-26T02:07:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:10 smithi072 ceph-mon[41619]: osdmap e264: 8 total, 8 up, 8 in 2024-03-26T02:07:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2927718344' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi072-82404-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi072-82404-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi072-82387-42"}]: dispatch 2024-03-26T02:07:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi072-82387-42"}]: dispatch 2024-03-26T02:07:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi072-82387-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi072-82404-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:07:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi072-82387-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:10 smithi072 ceph-mon[41619]: osdmap e265: 8 total, 8 up, 8 in 2024-03-26T02:07:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi072-82387-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi072-82387-42"}]: dispatch 2024-03-26T02:07:11.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:10 smithi072 ceph-mon[30177]: pgmap v355: 324 pgs: 32 unknown, 292 active+clean; 4.6 MiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:07:11.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-35", "mode": "writeback"}]': finished 2024-03-26T02:07:11.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:10 smithi072 ceph-mon[30177]: osdmap e264: 8 total, 8 up, 8 in 2024-03-26T02:07:11.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2927718344' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi072-82404-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:11.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi072-82404-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:11.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi072-82387-42"}]: dispatch 2024-03-26T02:07:11.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi072-82387-42"}]: dispatch 2024-03-26T02:07:11.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi072-82387-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:11.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi072-82404-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:07:11.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi072-82387-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:11.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:10 smithi072 ceph-mon[30177]: osdmap e265: 8 total, 8 up, 8 in 2024-03-26T02:07:11.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi072-82387-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi072-82387-42"}]: dispatch 2024-03-26T02:07:12.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:11 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:07:12.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:11 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:07:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:11 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:07:13.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[30177]: pgmap v358: 356 pgs: 32 creating+peering, 324 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:07:13.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[30177]: osdmap e266: 8 total, 8 up, 8 in 2024-03-26T02:07:13.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3806530472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:13.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:13.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3806530472' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:13.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:13.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3806530472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi072-82404-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:13.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi072-82404-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:13.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:13.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:13.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi072-82387-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi072-82387-42"}]': finished 2024-03-26T02:07:13.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi072-82404-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:13.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:07:13.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[30177]: osdmap e267: 8 total, 8 up, 8 in 2024-03-26T02:07:13.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3806530472' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi072-82404-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:13.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi072-82404-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:13.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-35"}]: dispatch 2024-03-26T02:07:13.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-35"}]: dispatch 2024-03-26T02:07:13.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[41619]: pgmap v358: 356 pgs: 32 creating+peering, 324 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:07:13.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[41619]: osdmap e266: 8 total, 8 up, 8 in 2024-03-26T02:07:13.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3806530472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:13.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:13.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3806530472' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:13.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:13.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3806530472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi072-82404-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:13.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi072-82404-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:13.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:13.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:13.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi072-82387-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi072-82387-42"}]': finished 2024-03-26T02:07:13.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi072-82404-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:13.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:07:13.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[41619]: osdmap e267: 8 total, 8 up, 8 in 2024-03-26T02:07:13.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3806530472' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi072-82404-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:13.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi072-82404-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:13.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-35"}]: dispatch 2024-03-26T02:07:13.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-35"}]: dispatch 2024-03-26T02:07:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:12 smithi173 ceph-mon[33839]: pgmap v358: 356 pgs: 32 creating+peering, 324 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:07:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:12 smithi173 ceph-mon[33839]: osdmap e266: 8 total, 8 up, 8 in 2024-03-26T02:07:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3806530472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:13.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3806530472' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:13.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:13.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3806530472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi072-82404-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:13.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi072-82404-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:13.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:13.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:13.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi072-82387-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi072-82387-42"}]': finished 2024-03-26T02:07:13.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi072-82404-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:13.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:07:13.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:12 smithi173 ceph-mon[33839]: osdmap e267: 8 total, 8 up, 8 in 2024-03-26T02:07:13.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3806530472' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi072-82404-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:13.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi072-82404-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:13.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-35"}]: dispatch 2024-03-26T02:07:13.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-35"}]: dispatch 2024-03-26T02:07:14.983 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:14 smithi072 ceph-mon[30177]: pgmap v361: 332 pgs: 8 unknown, 324 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:07:14.983 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:14 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-35"}]': finished 2024-03-26T02:07:14.983 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:14 smithi072 ceph-mon[30177]: osdmap e268: 8 total, 8 up, 8 in 2024-03-26T02:07:14.983 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:14 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:14.984 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:14 smithi072 ceph-mon[41619]: pgmap v361: 332 pgs: 8 unknown, 324 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:07:14.984 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:14 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-35"}]': finished 2024-03-26T02:07:14.984 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:14 smithi072 ceph-mon[41619]: osdmap e268: 8 total, 8 up, 8 in 2024-03-26T02:07:14.984 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:14 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:15.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:14 smithi173 ceph-mon[33839]: pgmap v361: 332 pgs: 8 unknown, 324 active+clean; 4.6 MiB data, 576 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:07:15.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:14 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-35"}]': finished 2024-03-26T02:07:15.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:14 smithi173 ceph-mon[33839]: osdmap e268: 8 total, 8 up, 8 in 2024-03-26T02:07:15.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:14 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:15.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:07:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:07:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:07:16.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi072-82404-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi072-82404-52"}]': finished 2024-03-26T02:07:16.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:15 smithi173 ceph-mon[33839]: osdmap e269: 8 total, 8 up, 8 in 2024-03-26T02:07:16.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi072-82387-42"}]: dispatch 2024-03-26T02:07:16.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:16.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi072-82404-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi072-82404-52"}]': finished 2024-03-26T02:07:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:15 smithi072 ceph-mon[30177]: osdmap e269: 8 total, 8 up, 8 in 2024-03-26T02:07:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi072-82387-42"}]: dispatch 2024-03-26T02:07:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:16.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi072-82404-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi072-82404-52"}]': finished 2024-03-26T02:07:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:15 smithi072 ceph-mon[41619]: osdmap e269: 8 total, 8 up, 8 in 2024-03-26T02:07:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi072-82387-42"}]: dispatch 2024-03-26T02:07:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:16.760 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAi api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6983 ms) 2024-03-26T02:07:16.760 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-03-26T02:07:16.761 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7003 ms) 2024-03-26T02:07:16.761 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-03-26T02:07:16.761 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (6996 ms) 2024-03-26T02:07:16.761 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-03-26T02:07:16.761 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7020 ms) 2024-03-26T02:07:16.761 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-03-26T02:07:16.761 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7015 ms) 2024-03-26T02:07:16.761 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-03-26T02:07:16.761 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7015 ms) 2024-03-26T02:07:16.761 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-03-26T02:07:16.761 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7012 ms) 2024-03-26T02:07:16.761 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-03-26T02:07:16.761 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (6946 ms) 2024-03-26T02:07:16.761 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-03-26T02:07:16.762 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7046 ms) 2024-03-26T02:07:16.762 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-03-26T02:07:16.762 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7032 ms) 2024-03-26T02:07:16.762 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-03-26T02:07:16.762 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6931 ms) 2024-03-26T02:07:16.762 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-03-26T02:07:16.762 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7014 ms) 2024-03-26T02:07:16.762 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-03-26T02:07:16.762 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6985 ms) 2024-03-26T02:07:16.762 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-03-26T02:07:16.762 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7038 ms) 2024-03-26T02:07:16.762 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-03-26T02:07:16.762 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7046 ms) 2024-03-26T02:07:16.763 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-03-26T02:07:16.763 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7032 ms) 2024-03-26T02:07:16.763 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (112114 ms total) 2024-03-26T02:07:16.763 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: 2024-03-26T02:07:16.763 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [----------] Global test environment tear-down 2024-03-26T02:07:16.763 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (207716 ms total) 2024-03-26T02:07:16.763 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio: [ PASSED ] 42 tests. 2024-03-26T02:07:17.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:16 smithi173 ceph-mon[33839]: pgmap v364: 332 pgs: 8 creating+peering, 324 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T02:07:17.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:16 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi072-82387-42"}]': finished 2024-03-26T02:07:17.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:16 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:07:17.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:16 smithi173 ceph-mon[33839]: osdmap e270: 8 total, 8 up, 8 in 2024-03-26T02:07:17.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:16 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-35"}]: dispatch 2024-03-26T02:07:17.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:16 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi072-82387-42"}]: dispatch 2024-03-26T02:07:17.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:16 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-35"}]: dispatch 2024-03-26T02:07:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:16 smithi072 ceph-mon[30177]: pgmap v364: 332 pgs: 8 creating+peering, 324 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T02:07:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:16 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi072-82387-42"}]': finished 2024-03-26T02:07:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:16 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:07:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:16 smithi072 ceph-mon[30177]: osdmap e270: 8 total, 8 up, 8 in 2024-03-26T02:07:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:16 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-35"}]: dispatch 2024-03-26T02:07:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:16 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi072-82387-42"}]: dispatch 2024-03-26T02:07:17.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:16 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-35"}]: dispatch 2024-03-26T02:07:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:16 smithi072 ceph-mon[41619]: pgmap v364: 332 pgs: 8 creating+peering, 324 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T02:07:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:16 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi072-82387-42"}]': finished 2024-03-26T02:07:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:16 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:07:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:16 smithi072 ceph-mon[41619]: osdmap e270: 8 total, 8 up, 8 in 2024-03-26T02:07:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:16 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-35"}]: dispatch 2024-03-26T02:07:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:16 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi072-82387-42"}]: dispatch 2024-03-26T02:07:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:16 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-35"}]: dispatch 2024-03-26T02:07:18.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:17 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:07:18.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:17 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi072-82387-42"}]': finished 2024-03-26T02:07:18.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:17 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-35"}]': finished 2024-03-26T02:07:18.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:17 smithi173 ceph-mon[33839]: osdmap e271: 8 total, 8 up, 8 in 2024-03-26T02:07:18.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:17 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3806530472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:18.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:17 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:18.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:17 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:07:18.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:17 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi072-82387-42"}]': finished 2024-03-26T02:07:18.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:17 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-35"}]': finished 2024-03-26T02:07:18.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:17 smithi072 ceph-mon[30177]: osdmap e271: 8 total, 8 up, 8 in 2024-03-26T02:07:18.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:17 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3806530472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:18.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:17 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:18.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:17 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:07:18.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:17 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2394301988' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi072-82387-42"}]': finished 2024-03-26T02:07:18.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:17 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-35"}]': finished 2024-03-26T02:07:18.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:17 smithi072 ceph-mon[41619]: osdmap e271: 8 total, 8 up, 8 in 2024-03-26T02:07:18.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:17 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3806530472' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:18.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:17 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:19.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:18 smithi173 ceph-mon[33839]: pgmap v367: 324 pgs: 324 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T02:07:19.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi072-82404-52"}]': finished 2024-03-26T02:07:19.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:18 smithi173 ceph-mon[33839]: osdmap e272: 8 total, 8 up, 8 in 2024-03-26T02:07:19.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3806530472' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:19.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:19.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:18 smithi072 ceph-mon[41619]: pgmap v367: 324 pgs: 324 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T02:07:19.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi072-82404-52"}]': finished 2024-03-26T02:07:19.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:18 smithi072 ceph-mon[41619]: osdmap e272: 8 total, 8 up, 8 in 2024-03-26T02:07:19.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3806530472' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:19.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:19.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:18 smithi072 ceph-mon[30177]: pgmap v367: 324 pgs: 324 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T02:07:19.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi072-82404-52"}]': finished 2024-03-26T02:07:19.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:18 smithi072 ceph-mon[30177]: osdmap e272: 8 total, 8 up, 8 in 2024-03-26T02:07:19.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3806530472' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:19.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi072-82404-52"}]: dispatch 2024-03-26T02:07:20.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:19 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi072-82404-52"}]': finished 2024-03-26T02:07:20.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:19 smithi173 ceph-mon[33839]: osdmap e273: 8 total, 8 up, 8 in 2024-03-26T02:07:20.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:20.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:19 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:20.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi072-82404-53"}]: dispatch 2024-03-26T02:07:20.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi072-82404-53"}]: dispatch 2024-03-26T02:07:20.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi072-82404-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:20.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:19 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:07:20.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi072-82404-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:20.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:19 smithi173 ceph-mon[33839]: osdmap e274: 8 total, 8 up, 8 in 2024-03-26T02:07:20.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi072-82404-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi072-82404-53"}]: dispatch 2024-03-26T02:07:20.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:20.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:19 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:20.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:19 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi072-82404-52"}]': finished 2024-03-26T02:07:20.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:19 smithi072 ceph-mon[30177]: osdmap e273: 8 total, 8 up, 8 in 2024-03-26T02:07:20.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:20.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:19 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:20.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi072-82404-53"}]: dispatch 2024-03-26T02:07:20.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi072-82404-53"}]: dispatch 2024-03-26T02:07:20.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi072-82404-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:20.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:19 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:07:20.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi072-82404-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:20.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:19 smithi072 ceph-mon[30177]: osdmap e274: 8 total, 8 up, 8 in 2024-03-26T02:07:20.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi072-82404-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi072-82404-53"}]: dispatch 2024-03-26T02:07:20.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:20.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:19 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:20.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:19 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi072-82404-52"}]': finished 2024-03-26T02:07:20.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:19 smithi072 ceph-mon[41619]: osdmap e273: 8 total, 8 up, 8 in 2024-03-26T02:07:20.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:20.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:19 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:20.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi072-82404-53"}]: dispatch 2024-03-26T02:07:20.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi072-82404-53"}]: dispatch 2024-03-26T02:07:20.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi072-82404-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:20.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:19 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:07:20.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi072-82404-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:20.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:19 smithi072 ceph-mon[41619]: osdmap e274: 8 total, 8 up, 8 in 2024-03-26T02:07:20.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi072-82404-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi072-82404-53"}]: dispatch 2024-03-26T02:07:20.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:20.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:19 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:21.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:07:20 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:07:21.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:20 smithi173 ceph-mon[33839]: pgmap v370: 324 pgs: 32 unknown, 292 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:07:21.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:20 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:21.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:20 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:07:21.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:20 smithi072 ceph-mon[30177]: pgmap v370: 324 pgs: 32 unknown, 292 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:07:21.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:20 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:21.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:20 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:07:21.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:20 smithi072 ceph-mon[41619]: pgmap v370: 324 pgs: 32 unknown, 292 active+clean; 4.6 MiB data, 581 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:07:21.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:20 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:21.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:20 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:07:22.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-37"}]: dispatch 2024-03-26T02:07:22.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:21 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:07:22.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:21 smithi072 ceph-mon[30177]: osdmap e275: 8 total, 8 up, 8 in 2024-03-26T02:07:22.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:21 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-37"}]: dispatch 2024-03-26T02:07:22.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:21 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:07:22.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi072-82404-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi072-82404-53"}]': finished 2024-03-26T02:07:22.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:21 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-37"}]': finished 2024-03-26T02:07:22.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:21 smithi072 ceph-mon[30177]: osdmap e276: 8 total, 8 up, 8 in 2024-03-26T02:07:22.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-37", "mode": "writeback"}]: dispatch 2024-03-26T02:07:22.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:21 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-37", "mode": "writeback"}]: dispatch 2024-03-26T02:07:22.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-37"}]: dispatch 2024-03-26T02:07:22.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:21 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:07:22.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:21 smithi072 ceph-mon[41619]: osdmap e275: 8 total, 8 up, 8 in 2024-03-26T02:07:22.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:21 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-37"}]: dispatch 2024-03-26T02:07:22.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:21 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:07:22.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi072-82404-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi072-82404-53"}]': finished 2024-03-26T02:07:22.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:21 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-37"}]': finished 2024-03-26T02:07:22.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:21 smithi072 ceph-mon[41619]: osdmap e276: 8 total, 8 up, 8 in 2024-03-26T02:07:22.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-37", "mode": "writeback"}]: dispatch 2024-03-26T02:07:22.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:21 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-37", "mode": "writeback"}]: dispatch 2024-03-26T02:07:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-37"}]: dispatch 2024-03-26T02:07:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:21 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:07:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:21 smithi173 ceph-mon[33839]: osdmap e275: 8 total, 8 up, 8 in 2024-03-26T02:07:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:21 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-37"}]: dispatch 2024-03-26T02:07:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:21 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:07:22.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi072-82404-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi072-82404-53"}]': finished 2024-03-26T02:07:22.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:21 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-37"}]': finished 2024-03-26T02:07:22.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:21 smithi173 ceph-mon[33839]: osdmap e276: 8 total, 8 up, 8 in 2024-03-26T02:07:22.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-37", "mode": "writeback"}]: dispatch 2024-03-26T02:07:22.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:21 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-37", "mode": "writeback"}]: dispatch 2024-03-26T02:07:23.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:22 smithi072 ceph-mon[41619]: pgmap v373: 324 pgs: 324 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T02:07:23.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:22 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:07:23.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:22 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-37", "mode": "writeback"}]': finished 2024-03-26T02:07:23.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:22 smithi072 ceph-mon[41619]: osdmap e277: 8 total, 8 up, 8 in 2024-03-26T02:07:23.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:22 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:23.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:22 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:23.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:22 smithi072 ceph-mon[30177]: pgmap v373: 324 pgs: 324 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T02:07:23.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:22 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:07:23.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:22 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-37", "mode": "writeback"}]': finished 2024-03-26T02:07:23.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:22 smithi072 ceph-mon[30177]: osdmap e277: 8 total, 8 up, 8 in 2024-03-26T02:07:23.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:22 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:23.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:22 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:22 smithi173 ceph-mon[33839]: pgmap v373: 324 pgs: 324 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T02:07:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:22 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:07:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:22 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-37", "mode": "writeback"}]': finished 2024-03-26T02:07:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:22 smithi173 ceph-mon[33839]: osdmap e277: 8 total, 8 up, 8 in 2024-03-26T02:07:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:22 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:22 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:24.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:24 smithi072 ceph-mon[30177]: pgmap v376: 332 pgs: 8 unknown, 324 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:24.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:24 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:07:24.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:24 smithi072 ceph-mon[30177]: osdmap e278: 8 total, 8 up, 8 in 2024-03-26T02:07:24.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:24 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-37"}]: dispatch 2024-03-26T02:07:24.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:24 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-37"}]: dispatch 2024-03-26T02:07:24.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:24 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi072-82404-53"}]: dispatch 2024-03-26T02:07:24.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:24 smithi072 ceph-mon[41619]: pgmap v376: 332 pgs: 8 unknown, 324 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:24.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:24 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:07:24.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:24 smithi072 ceph-mon[41619]: osdmap e278: 8 total, 8 up, 8 in 2024-03-26T02:07:24.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:24 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-37"}]: dispatch 2024-03-26T02:07:24.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:24 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-37"}]: dispatch 2024-03-26T02:07:24.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:24 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi072-82404-53"}]: dispatch 2024-03-26T02:07:25.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:24 smithi173 ceph-mon[33839]: pgmap v376: 332 pgs: 8 unknown, 324 active+clean; 4.6 MiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:25.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:24 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:07:25.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:24 smithi173 ceph-mon[33839]: osdmap e278: 8 total, 8 up, 8 in 2024-03-26T02:07:25.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:24 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-37"}]: dispatch 2024-03-26T02:07:25.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:24 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-37"}]: dispatch 2024-03-26T02:07:25.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:24 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi072-82404-53"}]: dispatch 2024-03-26T02:07:25.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:07:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:07:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:07:26.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:25 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:07:26.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:25 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-37"}]': finished 2024-03-26T02:07:26.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi072-82404-53"}]': finished 2024-03-26T02:07:26.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:25 smithi072 ceph-mon[41619]: osdmap e279: 8 total, 8 up, 8 in 2024-03-26T02:07:26.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi072-82404-53"}]: dispatch 2024-03-26T02:07:26.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi072-82404-53"}]': finished 2024-03-26T02:07:26.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:25 smithi072 ceph-mon[41619]: osdmap e280: 8 total, 8 up, 8 in 2024-03-26T02:07:26.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:25 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:07:26.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:25 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-37"}]': finished 2024-03-26T02:07:26.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi072-82404-53"}]': finished 2024-03-26T02:07:26.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:25 smithi072 ceph-mon[30177]: osdmap e279: 8 total, 8 up, 8 in 2024-03-26T02:07:26.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi072-82404-53"}]: dispatch 2024-03-26T02:07:26.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi072-82404-53"}]': finished 2024-03-26T02:07:26.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:25 smithi072 ceph-mon[30177]: osdmap e280: 8 total, 8 up, 8 in 2024-03-26T02:07:26.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:25 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:07:26.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:25 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-37"}]': finished 2024-03-26T02:07:26.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi072-82404-53"}]': finished 2024-03-26T02:07:26.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:25 smithi173 ceph-mon[33839]: osdmap e279: 8 total, 8 up, 8 in 2024-03-26T02:07:26.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi072-82404-53"}]: dispatch 2024-03-26T02:07:26.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/954091184' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi072-82404-53"}]': finished 2024-03-26T02:07:26.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:25 smithi173 ceph-mon[33839]: osdmap e280: 8 total, 8 up, 8 in 2024-03-26T02:07:27.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[41619]: pgmap v379: 324 pgs: 324 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-03-26T02:07:27.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:27.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4066316038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:27.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4066316038' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4066316038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi072-82404-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi072-82404-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi072-82404-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[41619]: osdmap e281: 8 total, 8 up, 8 in 2024-03-26T02:07:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4066316038' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi072-82404-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi072-82404-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:27.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[30177]: pgmap v379: 324 pgs: 324 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-03-26T02:07:27.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:27.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4066316038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:27.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:27.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4066316038' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:27.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:27.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4066316038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi072-82404-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:27.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi072-82404-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:27.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi072-82404-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:27.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[30177]: osdmap e281: 8 total, 8 up, 8 in 2024-03-26T02:07:27.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4066316038' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi072-82404-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:27.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:27.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi072-82404-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:27.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:26 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:27.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:26 smithi173 ceph-mon[33839]: pgmap v379: 324 pgs: 324 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-03-26T02:07:27.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:26 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:27.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:26 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4066316038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:27.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:26 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:27.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:26 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4066316038' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:27.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:26 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:27.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:26 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4066316038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi072-82404-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:27.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:26 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi072-82404-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:27.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:26 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi072-82404-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:27.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:26 smithi173 ceph-mon[33839]: osdmap e281: 8 total, 8 up, 8 in 2024-03-26T02:07:27.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:26 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4066316038' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi072-82404-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:27.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:26 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:27.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:26 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi072-82404-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:27.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:26 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:28.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:28 smithi072 ceph-mon[41619]: pgmap v382: 324 pgs: 32 unknown, 292 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T02:07:28.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:28 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:07:28.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:28 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:28.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:28 smithi072 ceph-mon[41619]: osdmap e282: 8 total, 8 up, 8 in 2024-03-26T02:07:28.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:28 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:28.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:28 smithi072 ceph-mon[30177]: pgmap v382: 324 pgs: 32 unknown, 292 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T02:07:28.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:28 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:07:28.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:28 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:28.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:28 smithi072 ceph-mon[30177]: osdmap e282: 8 total, 8 up, 8 in 2024-03-26T02:07:28.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:28 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:29.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:28 smithi173 ceph-mon[33839]: pgmap v382: 324 pgs: 32 unknown, 292 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T02:07:29.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:28 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:07:29.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:28 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:29.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:28 smithi173 ceph-mon[33839]: osdmap e282: 8 total, 8 up, 8 in 2024-03-26T02:07:29.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:28 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:29.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:29 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi072-82404-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi072-82404-54"}]': finished 2024-03-26T02:07:29.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:29 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:07:29.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:29 smithi072 ceph-mon[30177]: osdmap e283: 8 total, 8 up, 8 in 2024-03-26T02:07:29.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:29 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-39"}]: dispatch 2024-03-26T02:07:29.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:29 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-39"}]: dispatch 2024-03-26T02:07:29.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:29 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi072-82404-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi072-82404-54"}]': finished 2024-03-26T02:07:29.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:29 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:07:29.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:29 smithi072 ceph-mon[41619]: osdmap e283: 8 total, 8 up, 8 in 2024-03-26T02:07:29.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:29 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-39"}]: dispatch 2024-03-26T02:07:29.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:29 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-39"}]: dispatch 2024-03-26T02:07:30.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:29 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi072-82404-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi072-82404-54"}]': finished 2024-03-26T02:07:30.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:29 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:07:30.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:29 smithi173 ceph-mon[33839]: osdmap e283: 8 total, 8 up, 8 in 2024-03-26T02:07:30.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:29 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-39"}]: dispatch 2024-03-26T02:07:30.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:29 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-39"}]: dispatch 2024-03-26T02:07:30.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:30 smithi072 ceph-mon[30177]: pgmap v385: 332 pgs: 40 unknown, 292 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:07:30.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-39"}]': finished 2024-03-26T02:07:30.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:30 smithi072 ceph-mon[30177]: osdmap e284: 8 total, 8 up, 8 in 2024-03-26T02:07:30.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-39", "mode": "writeback"}]: dispatch 2024-03-26T02:07:30.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-39", "mode": "writeback"}]: dispatch 2024-03-26T02:07:30.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:30 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:07:30.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-39", "mode": "writeback"}]': finished 2024-03-26T02:07:30.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:30 smithi072 ceph-mon[30177]: osdmap e285: 8 total, 8 up, 8 in 2024-03-26T02:07:30.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:30 smithi072 ceph-mon[41619]: pgmap v385: 332 pgs: 40 unknown, 292 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:07:30.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-39"}]': finished 2024-03-26T02:07:30.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:30 smithi072 ceph-mon[41619]: osdmap e284: 8 total, 8 up, 8 in 2024-03-26T02:07:30.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-39", "mode": "writeback"}]: dispatch 2024-03-26T02:07:30.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-39", "mode": "writeback"}]: dispatch 2024-03-26T02:07:30.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:30 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:07:30.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-39", "mode": "writeback"}]': finished 2024-03-26T02:07:30.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:30 smithi072 ceph-mon[41619]: osdmap e285: 8 total, 8 up, 8 in 2024-03-26T02:07:31.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:07:30 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:07:31.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:30 smithi173 ceph-mon[33839]: pgmap v385: 332 pgs: 40 unknown, 292 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:07:31.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-39"}]': finished 2024-03-26T02:07:31.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:30 smithi173 ceph-mon[33839]: osdmap e284: 8 total, 8 up, 8 in 2024-03-26T02:07:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-39", "mode": "writeback"}]: dispatch 2024-03-26T02:07:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-39", "mode": "writeback"}]: dispatch 2024-03-26T02:07:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:30 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:07:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-39", "mode": "writeback"}]': finished 2024-03-26T02:07:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:30 smithi173 ceph-mon[33839]: osdmap e285: 8 total, 8 up, 8 in 2024-03-26T02:07:31.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:31 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4066316038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:31.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:31 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:31.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:31 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:07:31.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:31 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:31.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:31 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:31.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:31 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi072-82404-54"}]': finished 2024-03-26T02:07:31.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:31 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:07:31.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:31 smithi072 ceph-mon[30177]: osdmap e286: 8 total, 8 up, 8 in 2024-03-26T02:07:31.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:31 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-39"}]: dispatch 2024-03-26T02:07:31.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:31 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4066316038' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:31.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:31 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-39"}]: dispatch 2024-03-26T02:07:31.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:31 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:31.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:31 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4066316038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:31.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:31 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:31.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:31 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:07:31.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:31 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:31.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:31 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:31.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:31 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi072-82404-54"}]': finished 2024-03-26T02:07:31.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:31 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:07:31.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:31 smithi072 ceph-mon[41619]: osdmap e286: 8 total, 8 up, 8 in 2024-03-26T02:07:31.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:31 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-39"}]: dispatch 2024-03-26T02:07:31.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:31 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4066316038' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:31.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:31 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-39"}]: dispatch 2024-03-26T02:07:31.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:31 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:31 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4066316038' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:31 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:31 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:07:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:31 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:31 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:32.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:31 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi072-82404-54"}]': finished 2024-03-26T02:07:32.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:31 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:07:32.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:31 smithi173 ceph-mon[33839]: osdmap e286: 8 total, 8 up, 8 in 2024-03-26T02:07:32.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:31 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-39"}]: dispatch 2024-03-26T02:07:32.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:31 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4066316038' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:32.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:31 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-39"}]: dispatch 2024-03-26T02:07:32.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:31 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi072-82404-54"}]: dispatch 2024-03-26T02:07:32.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:32 smithi072 ceph-mon[30177]: pgmap v388: 324 pgs: 324 active+clean; 4.6 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:32.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:32 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:32.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:32 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:07:32.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:32 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-39"}]': finished 2024-03-26T02:07:32.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:32 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi072-82404-54"}]': finished 2024-03-26T02:07:32.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:32 smithi072 ceph-mon[30177]: osdmap e287: 8 total, 8 up, 8 in 2024-03-26T02:07:32.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:32 smithi072 ceph-mon[41619]: pgmap v388: 324 pgs: 324 active+clean; 4.6 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:32.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:32 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:32.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:32 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:07:32.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:32 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-39"}]': finished 2024-03-26T02:07:32.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:32 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi072-82404-54"}]': finished 2024-03-26T02:07:32.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:32 smithi072 ceph-mon[41619]: osdmap e287: 8 total, 8 up, 8 in 2024-03-26T02:07:33.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:32 smithi173 ceph-mon[33839]: pgmap v388: 324 pgs: 324 active+clean; 4.6 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:33.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:32 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:33.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:32 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:07:33.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:32 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-39"}]': finished 2024-03-26T02:07:33.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:32 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi072-82404-54"}]': finished 2024-03-26T02:07:33.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:32 smithi173 ceph-mon[33839]: osdmap e287: 8 total, 8 up, 8 in 2024-03-26T02:07:33.695 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-03-26T02:07:33.695 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-03-26T02:07:33.695 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-03-26T02:07:33.695 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: seed 82692 2024-03-26T02:07:33.695 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-03-26T02:07:33.696 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (342 ms) 2024-03-26T02:07:33.696 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-03-26T02:07:33.696 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (16993 ms) 2024-03-26T02:07:33.696 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-03-26T02:07:33.696 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (56 ms) 2024-03-26T02:07:33.696 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (17391 ms total) 2024-03-26T02:07:33.696 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: 2024-03-26T02:07:33.696 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-03-26T02:07:33.696 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-03-26T02:07:33.696 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (8759 ms) 2024-03-26T02:07:33.696 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-03-26T02:07:33.696 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (7901 ms) 2024-03-26T02:07:33.696 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-03-26T02:07:33.697 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (9804 ms) 2024-03-26T02:07:33.697 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-03-26T02:07:33.697 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: my_snaps [3] 2024-03-26T02:07:33.697 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: my_snaps [4,3] 2024-03-26T02:07:33.697 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: my_snaps [5,4,3] 2024-03-26T02:07:33.697 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-03-26T02:07:33.697 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: promoting some heads 2024-03-26T02:07:33.697 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: promoting from clones for snap 6 2024-03-26T02:07:33.697 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: promoting from clones for snap 5 2024-03-26T02:07:33.697 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: promoting from clones for snap 4 2024-03-26T02:07:33.697 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: promoting from clones for snap 3 2024-03-26T02:07:33.697 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: waiting for scrubs... 2024-03-26T02:07:33.697 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: done waiting 2024-03-26T02:07:33.697 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (46010 ms) 2024-03-26T02:07:33.698 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-03-26T02:07:33.698 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10040 ms) 2024-03-26T02:07:33.698 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-03-26T02:07:33.698 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (7971 ms) 2024-03-26T02:07:33.698 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-03-26T02:07:33.698 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8023 ms) 2024-03-26T02:07:33.698 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-03-26T02:07:33.698 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8015 ms) 2024-03-26T02:07:33.698 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-03-26T02:07:33.698 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10019 ms) 2024-03-26T02:07:33.698 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-03-26T02:07:33.698 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8976 ms) 2024-03-26T02:07:33.698 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-03-26T02:07:33.698 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10284 ms) 2024-03-26T02:07:33.699 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-03-26T02:07:33.699 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (12704 ms) 2024-03-26T02:07:33.699 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-03-26T02:07:33.699 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (8224 ms) 2024-03-26T02:07:33.699 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-03-26T02:07:33.699 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (8054 ms) 2024-03-26T02:07:33.699 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-03-26T02:07:33.699 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (12827 ms) 2024-03-26T02:07:33.699 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-03-26T02:07:33.699 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7895 ms) 2024-03-26T02:07:33.699 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-03-26T02:07:33.699 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8039 ms) 2024-03-26T02:07:33.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:33 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1212842460' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:33.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:33 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:33.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:33 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1212842460' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:33.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:33 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:33.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:33 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1212842460' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi072-82404-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:33.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:33 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi072-82404-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:33.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:33 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1212842460' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:33.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:33 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:33.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:33 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1212842460' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:33.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:33 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:33.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:33 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1212842460' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi072-82404-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:33.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:33 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi072-82404-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:34.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:33 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1212842460' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:33 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:33 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1212842460' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:33 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:33 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1212842460' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi072-82404-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:34.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:33 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi072-82404-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:35.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:34 smithi072 ceph-mon[41619]: pgmap v391: 324 pgs: 324 active+clean; 4.6 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:35.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:34 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi072-82404-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:35.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:34 smithi072 ceph-mon[41619]: osdmap e288: 8 total, 8 up, 8 in 2024-03-26T02:07:35.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:34 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1212842460' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi072-82404-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:35.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:34 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi072-82404-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:35.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:34 smithi072 ceph-mon[30177]: pgmap v391: 324 pgs: 324 active+clean; 4.6 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:35.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:34 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi072-82404-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:35.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:34 smithi072 ceph-mon[30177]: osdmap e288: 8 total, 8 up, 8 in 2024-03-26T02:07:35.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:34 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1212842460' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi072-82404-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:35.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:34 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi072-82404-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:35.220 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:07:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:07:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:07:35.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:34 smithi173 ceph-mon[33839]: pgmap v391: 324 pgs: 324 active+clean; 4.6 MiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:35.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:34 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi072-82404-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:35.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:34 smithi173 ceph-mon[33839]: osdmap e288: 8 total, 8 up, 8 in 2024-03-26T02:07:35.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:34 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1212842460' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi072-82404-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:35.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:34 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi072-82404-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:35 smithi072 ceph-mon[30177]: osdmap e289: 8 total, 8 up, 8 in 2024-03-26T02:07:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:35 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:35 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:35 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi072-82404-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi072-82404-55"}]': finished 2024-03-26T02:07:36.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:35 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:07:36.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:35 smithi072 ceph-mon[30177]: osdmap e290: 8 total, 8 up, 8 in 2024-03-26T02:07:36.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:35 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:36.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:35 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:36.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:35 smithi072 ceph-mon[41619]: osdmap e289: 8 total, 8 up, 8 in 2024-03-26T02:07:36.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:35 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:36.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:35 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:36.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:35 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi072-82404-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi072-82404-55"}]': finished 2024-03-26T02:07:36.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:35 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:07:36.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:35 smithi072 ceph-mon[41619]: osdmap e290: 8 total, 8 up, 8 in 2024-03-26T02:07:36.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:35 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:36.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:35 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:36.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:35 smithi173 ceph-mon[33839]: osdmap e289: 8 total, 8 up, 8 in 2024-03-26T02:07:36.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:35 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:36.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:35 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:36.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:35 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi072-82404-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi072-82404-55"}]': finished 2024-03-26T02:07:36.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:35 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:07:36.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:35 smithi173 ceph-mon[33839]: osdmap e290: 8 total, 8 up, 8 in 2024-03-26T02:07:36.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:35 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:36.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:35 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:36 smithi072 ceph-mon[30177]: pgmap v394: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-26T02:07:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:36 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:07:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:36 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:07:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:36 smithi072 ceph-mon[30177]: osdmap e291: 8 total, 8 up, 8 in 2024-03-26T02:07:37.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:36 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:07:37.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:36 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:07:37.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:36 smithi072 ceph-mon[41619]: pgmap v394: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-26T02:07:37.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:36 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:07:37.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:36 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:07:37.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:36 smithi072 ceph-mon[41619]: osdmap e291: 8 total, 8 up, 8 in 2024-03-26T02:07:37.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:36 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:07:37.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:36 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:07:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:36 smithi173 ceph-mon[33839]: pgmap v394: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-26T02:07:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:36 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:07:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:36 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:07:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:36 smithi173 ceph-mon[33839]: osdmap e291: 8 total, 8 up, 8 in 2024-03-26T02:07:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:36 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:07:37.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:36 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:07:38.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:38 smithi072 ceph-mon[30177]: pgmap v397: 332 pgs: 8 unknown, 32 creating+peering, 292 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-26T02:07:38.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:38 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_count","val": "2"}]': finished 2024-03-26T02:07:38.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:38 smithi072 ceph-mon[30177]: osdmap e292: 8 total, 8 up, 8 in 2024-03-26T02:07:38.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:38 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:07:38.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:38 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1212842460' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:38.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:38 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:07:38.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:38 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:38.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:38 smithi072 ceph-mon[41619]: pgmap v397: 332 pgs: 8 unknown, 32 creating+peering, 292 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-26T02:07:38.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:38 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_count","val": "2"}]': finished 2024-03-26T02:07:38.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:38 smithi072 ceph-mon[41619]: osdmap e292: 8 total, 8 up, 8 in 2024-03-26T02:07:38.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:38 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:07:38.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:38 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1212842460' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:38.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:38 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:07:38.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:38 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:39.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:38 smithi173 ceph-mon[33839]: pgmap v397: 332 pgs: 8 unknown, 32 creating+peering, 292 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-26T02:07:39.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:38 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_count","val": "2"}]': finished 2024-03-26T02:07:39.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:38 smithi173 ceph-mon[33839]: osdmap e292: 8 total, 8 up, 8 in 2024-03-26T02:07:39.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:38 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:07:39.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:38 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1212842460' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:39.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:38 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:07:39.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:38 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:39.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:39 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:07:39.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:39 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi072-82404-55"}]': finished 2024-03-26T02:07:39.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:39 smithi072 ceph-mon[41619]: osdmap e293: 8 total, 8 up, 8 in 2024-03-26T02:07:39.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T02:07:39.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1212842460' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:39.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:39 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T02:07:39.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:39 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:39.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:39 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T02:07:39.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:39 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi072-82404-55"}]': finished 2024-03-26T02:07:39.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:39 smithi072 ceph-mon[41619]: osdmap e294: 8 total, 8 up, 8 in 2024-03-26T02:07:39.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:39 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:07:39.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:39 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi072-82404-55"}]': finished 2024-03-26T02:07:39.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:39 smithi072 ceph-mon[30177]: osdmap e293: 8 total, 8 up, 8 in 2024-03-26T02:07:39.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T02:07:39.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1212842460' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:39.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:39 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T02:07:39.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:39 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:39.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:39 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T02:07:39.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:39 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi072-82404-55"}]': finished 2024-03-26T02:07:39.971 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:39 smithi072 ceph-mon[30177]: osdmap e294: 8 total, 8 up, 8 in 2024-03-26T02:07:40.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:39 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:07:40.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:39 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi072-82404-55"}]': finished 2024-03-26T02:07:40.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:39 smithi173 ceph-mon[33839]: osdmap e293: 8 total, 8 up, 8 in 2024-03-26T02:07:40.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T02:07:40.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1212842460' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:40.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:39 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T02:07:40.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:39 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi072-82404-55"}]: dispatch 2024-03-26T02:07:40.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:39 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T02:07:40.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:39 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi072-82404-55"}]': finished 2024-03-26T02:07:40.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:39 smithi173 ceph-mon[33839]: osdmap e294: 8 total, 8 up, 8 in 2024-03-26T02:07:40.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:40 smithi072 ceph-mon[30177]: pgmap v400: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:07:40.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:40 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:40.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1935680936' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:40.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:40 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:40.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1935680936' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:40.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:40 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:40.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1935680936' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi072-82404-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:40.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:40 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi072-82404-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:40.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:40 smithi072 ceph-mon[41619]: pgmap v400: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:07:40.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:40 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:40.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1935680936' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:40.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:40 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:40.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1935680936' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:40.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:40 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:40.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1935680936' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi072-82404-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:40.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:40 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi072-82404-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:41.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:07:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:07:41.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:40 smithi173 ceph-mon[33839]: pgmap v400: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:07:41.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:40 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:41.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1935680936' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:41.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:40 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:41.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1935680936' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:41.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:40 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:41.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1935680936' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi072-82404-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:41.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:40 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi072-82404-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:42.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:41 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:42.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:41 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi072-82404-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:42.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:41 smithi173 ceph-mon[33839]: osdmap e295: 8 total, 8 up, 8 in 2024-03-26T02:07:42.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:41 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1935680936' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi072-82404-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:42.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:41 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:42.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:41 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi072-82404-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:42.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:41 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-41"}]: dispatch 2024-03-26T02:07:42.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:41 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-41"}]: dispatch 2024-03-26T02:07:42.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:41 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:07:42.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:41 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:42.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:41 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:07:42.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:41 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:07:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:41 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:41 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi072-82404-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:41 smithi072 ceph-mon[30177]: osdmap e295: 8 total, 8 up, 8 in 2024-03-26T02:07:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:41 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1935680936' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi072-82404-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:41 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:41 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi072-82404-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:41 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-41"}]: dispatch 2024-03-26T02:07:42.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:41 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-41"}]: dispatch 2024-03-26T02:07:42.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:41 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:07:42.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:41 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:42.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:41 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:07:42.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:41 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:07:42.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:41 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:42.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:41 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi072-82404-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:42.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:41 smithi072 ceph-mon[41619]: osdmap e295: 8 total, 8 up, 8 in 2024-03-26T02:07:42.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:41 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1935680936' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi072-82404-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:42.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:41 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:42.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:41 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi072-82404-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:42.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:41 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-41"}]: dispatch 2024-03-26T02:07:42.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:41 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-41"}]: dispatch 2024-03-26T02:07:42.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:41 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:07:42.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:41 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:07:42.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:41 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:07:42.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:41 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:07:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:42 smithi173 ceph-mon[33839]: pgmap v403: 324 pgs: 324 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:42 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-41"}]': finished 2024-03-26T02:07:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:42 smithi173 ceph-mon[33839]: osdmap e296: 8 total, 8 up, 8 in 2024-03-26T02:07:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:42 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:07:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:42 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:07:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:42 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:07:43.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:42 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi072-82404-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi072-82404-56"}]': finished 2024-03-26T02:07:43.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:42 smithi173 ceph-mon[33839]: osdmap e297: 8 total, 8 up, 8 in 2024-03-26T02:07:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:42 smithi072 ceph-mon[30177]: pgmap v403: 324 pgs: 324 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:42 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-41"}]': finished 2024-03-26T02:07:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:42 smithi072 ceph-mon[30177]: osdmap e296: 8 total, 8 up, 8 in 2024-03-26T02:07:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:42 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:07:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:42 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:07:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:42 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:07:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:42 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi072-82404-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi072-82404-56"}]': finished 2024-03-26T02:07:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:42 smithi072 ceph-mon[30177]: osdmap e297: 8 total, 8 up, 8 in 2024-03-26T02:07:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:42 smithi072 ceph-mon[41619]: pgmap v403: 324 pgs: 324 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:42 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-41"}]': finished 2024-03-26T02:07:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:42 smithi072 ceph-mon[41619]: osdmap e296: 8 total, 8 up, 8 in 2024-03-26T02:07:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:42 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:07:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:42 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:07:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:42 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:07:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:42 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi072-82404-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi072-82404-56"}]': finished 2024-03-26T02:07:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:42 smithi072 ceph-mon[41619]: osdmap e297: 8 total, 8 up, 8 in 2024-03-26T02:07:45.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:44 smithi173 ceph-mon[33839]: pgmap v406: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:45.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:44 smithi173 ceph-mon[33839]: osdmap e298: 8 total, 8 up, 8 in 2024-03-26T02:07:45.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:44 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:45.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:44 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:45.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:07:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:07:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:07:45.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:44 smithi072 ceph-mon[30177]: pgmap v406: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:45.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:44 smithi072 ceph-mon[30177]: osdmap e298: 8 total, 8 up, 8 in 2024-03-26T02:07:45.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:44 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:45.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:44 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:45.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:44 smithi072 ceph-mon[41619]: pgmap v406: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:45.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:44 smithi072 ceph-mon[41619]: osdmap e298: 8 total, 8 up, 8 in 2024-03-26T02:07:45.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:44 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:45.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:44 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:46.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:45 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:07:46.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:45 smithi173 ceph-mon[33839]: osdmap e299: 8 total, 8 up, 8 in 2024-03-26T02:07:46.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:45 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi072-82692-6","var": "pg_num","format": "json"}]: dispatch 2024-03-26T02:07:46.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:45 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:46.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:45 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1935680936' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:46.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:45 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:46.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:45 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:46.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:45 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:46.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:45 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:07:46.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:45 smithi072 ceph-mon[30177]: osdmap e299: 8 total, 8 up, 8 in 2024-03-26T02:07:46.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:45 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi072-82692-6","var": "pg_num","format": "json"}]: dispatch 2024-03-26T02:07:46.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:45 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:46.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:45 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1935680936' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:46.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:45 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:46.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:45 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:46.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:45 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:45 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:07:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:45 smithi072 ceph-mon[41619]: osdmap e299: 8 total, 8 up, 8 in 2024-03-26T02:07:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:45 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi072-82692-6","var": "pg_num","format": "json"}]: dispatch 2024-03-26T02:07:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:45 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:45 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1935680936' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:45 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:45 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:45 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:47.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:46 smithi173 ceph-mon[33839]: pgmap v409: 324 pgs: 324 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:47.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:46 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:07:47.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:46 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi072-82404-56"}]': finished 2024-03-26T02:07:47.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:46 smithi173 ceph-mon[33839]: osdmap e300: 8 total, 8 up, 8 in 2024-03-26T02:07:47.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:46 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1935680936' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:47.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:46 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T02:07:47.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:46 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:47.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:46 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T02:07:47.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:46 smithi072 ceph-mon[30177]: pgmap v409: 324 pgs: 324 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:47.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:46 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:07:47.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:46 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi072-82404-56"}]': finished 2024-03-26T02:07:47.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:46 smithi072 ceph-mon[30177]: osdmap e300: 8 total, 8 up, 8 in 2024-03-26T02:07:47.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:46 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1935680936' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:47.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:46 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T02:07:47.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:46 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:47.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:46 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T02:07:47.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:46 smithi072 ceph-mon[41619]: pgmap v409: 324 pgs: 324 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:47.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:46 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:07:47.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:46 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi072-82404-56"}]': finished 2024-03-26T02:07:47.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:46 smithi072 ceph-mon[41619]: osdmap e300: 8 total, 8 up, 8 in 2024-03-26T02:07:47.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:46 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1935680936' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:47.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:46 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T02:07:47.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:46 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi072-82404-56"}]: dispatch 2024-03-26T02:07:47.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:46 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T02:07:48.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:47 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi072-82404-56"}]': finished 2024-03-26T02:07:48.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:47 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_count","val": "8"}]': finished 2024-03-26T02:07:48.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:47 smithi173 ceph-mon[33839]: osdmap e301: 8 total, 8 up, 8 in 2024-03-26T02:07:48.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:47 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:07:48.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:47 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:07:48.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:47 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi072-82404-57"}]: dispatch 2024-03-26T02:07:48.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:47 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi072-82404-57"}]: dispatch 2024-03-26T02:07:48.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:47 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi072-82404-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:48.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:47 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi072-82404-56"}]': finished 2024-03-26T02:07:48.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:47 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_count","val": "8"}]': finished 2024-03-26T02:07:48.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:47 smithi072 ceph-mon[30177]: osdmap e301: 8 total, 8 up, 8 in 2024-03-26T02:07:48.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:47 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:07:48.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:47 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:07:48.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:47 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi072-82404-57"}]: dispatch 2024-03-26T02:07:48.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:47 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi072-82404-57"}]: dispatch 2024-03-26T02:07:48.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:47 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi072-82404-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:48.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:47 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi072-82404-56"}]': finished 2024-03-26T02:07:48.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:47 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_count","val": "8"}]': finished 2024-03-26T02:07:48.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:47 smithi072 ceph-mon[41619]: osdmap e301: 8 total, 8 up, 8 in 2024-03-26T02:07:48.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:47 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:07:48.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:47 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:07:48.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:47 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi072-82404-57"}]: dispatch 2024-03-26T02:07:48.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:47 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi072-82404-57"}]: dispatch 2024-03-26T02:07:48.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:47 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi072-82404-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:49.004 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-03-26T02:07:49.004 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-26T02:07:49.005 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: ok, hit_set contains 263:602f83fe:::foo:head 2024-03-26T02:07:49.005 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9042 ms) 2024-03-26T02:07:49.005 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-03-26T02:07:49.005 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg_num = 32 2024-03-26T02:07:49.005 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 0 ls 1711418869,0 2024-03-26T02:07:49.005 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 1 ls 1711418869,0 2024-03-26T02:07:49.005 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 2 ls 1711418869,0 2024-03-26T02:07:49.005 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 3 ls 1711418869,0 2024-03-26T02:07:49.005 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 4 ls 1711418869,0 2024-03-26T02:07:49.005 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 5 ls 1711418869,0 2024-03-26T02:07:49.005 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 6 ls 1711418869,0 2024-03-26T02:07:49.005 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 7 ls 1711418869,0 2024-03-26T02:07:49.005 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 8 ls 1711418869,0 2024-03-26T02:07:49.006 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 9 ls 1711418869,0 2024-03-26T02:07:49.006 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 10 ls 1711418869,0 2024-03-26T02:07:49.006 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 11 ls 1711418869,0 2024-03-26T02:07:49.006 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 12 ls 1711418869,0 2024-03-26T02:07:49.006 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 13 ls 1711418869,0 2024-03-26T02:07:49.006 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 14 ls 1711418869,0 2024-03-26T02:07:49.006 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 15 ls 1711418869,0 2024-03-26T02:07:49.006 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 16 ls 1711418869,0 2024-03-26T02:07:49.006 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 17 ls 1711418869,0 2024-03-26T02:07:49.006 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 18 ls 1711418869,0 2024-03-26T02:07:49.006 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 19 ls 1711418869,0 2024-03-26T02:07:49.006 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 20 ls 1711418869,0 2024-03-26T02:07:49.006 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 21 ls 1711418869,0 2024-03-26T02:07:49.006 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 22 ls 1711418869,0 2024-03-26T02:07:49.007 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 23 ls 1711418869,0 2024-03-26T02:07:49.007 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 24 ls 1711418869,0 2024-03-26T02:07:49.007 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 25 ls 1711418869,0 2024-03-26T02:07:49.007 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 26 ls 1711418869,0 2024-03-26T02:07:49.007 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 27 ls 1711418869,0 2024-03-26T02:07:49.007 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 28 ls 1711418869,0 2024-03-26T02:07:49.007 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 29 ls 1711418869,0 2024-03-26T02:07:49.007 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 30 ls 1711418869,0 2024-03-26T02:07:49.007 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg 31 ls 1711418869,0 2024-03-26T02:07:49.007 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: pg_num = 32 2024-03-26T02:07:49.007 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:6cac518f:::0:head 2024-03-26T02:07:49.007 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:02547ec2:::1:head 2024-03-26T02:07:49.007 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:f905c69b:::2:head 2024-03-26T02:07:49.007 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:cfc208b3:::3:head 2024-03-26T02:07:49.008 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:d83876eb:::4:head 2024-03-26T02:07:49.008 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:b29083e3:::5:head 2024-03-26T02:07:49.008 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:c4fdafeb:::6:head 2024-03-26T02:07:49.008 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:5c6b0b28:::7:head 2024-03-26T02:07:49.008 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:bd63b0f1:::8:head 2024-03-26T02:07:49.008 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:e960b815:::9:head 2024-03-26T02:07:49.008 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:52ea6a34:::10:head 2024-03-26T02:07:49.008 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:89d3ae78:::11:head 2024-03-26T02:07:49.008 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:de5d7c5f:::12:head 2024-03-26T02:07:49.008 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:566253c9:::13:head 2024-03-26T02:07:49.008 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:62a1935d:::14:head 2024-03-26T02:07:49.008 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:863748b0:::15:head 2024-03-26T02:07:49.008 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:3958e169:::16:head 2024-03-26T02:07:49.008 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:4d4dabf9:::17:head 2024-03-26T02:07:49.008 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:8391935d:::18:head 2024-03-26T02:07:49.009 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:28883081:::19:head 2024-03-26T02:07:49.009 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:69259c59:::20:head 2024-03-26T02:07:49.009 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:4bdb80b7:::21:head 2024-03-26T02:07:49.009 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:a11c5d71:::22:head 2024-03-26T02:07:49.009 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:271af37b:::23:head 2024-03-26T02:07:49.009 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:95b121be:::24:head 2024-03-26T02:07:49.009 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:58d1031b:::25:head 2024-03-26T02:07:49.009 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:0a050783:::26:head 2024-03-26T02:07:49.009 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:c709704c:::27:head 2024-03-26T02:07:49.009 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:cbe56eaf:::28:head 2024-03-26T02:07:49.009 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:86b4b162:::29:head 2024-03-26T02:07:49.009 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:70d89383:::30:head 2024-03-26T02:07:49.009 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:dd450c7c:::31:head 2024-03-26T02:07:49.010 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:6d5729b1:::32:head 2024-03-26T02:07:49.010 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:c388f3fb:::33:head 2024-03-26T02:07:49.010 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:56cfea31:::34:head 2024-03-26T02:07:49.010 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:9dbc1bf7:::35:head 2024-03-26T02:07:49.010 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:40b74ccd:::36:head 2024-03-26T02:07:49.010 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:4d5aaf42:::37:head 2024-03-26T02:07:49.010 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:920f362c:::38:head 2024-03-26T02:07:49.010 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:6cc53222:::39:head 2024-03-26T02:07:49.010 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:9cad833f:::40:head 2024-03-26T02:07:49.010 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:1ea84d41:::41:head 2024-03-26T02:07:49.010 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:c4480ef6:::42:head 2024-03-26T02:07:49.010 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:a694361e:::43:head 2024-03-26T02:07:49.010 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:d1bd33e9:::44:head 2024-03-26T02:07:49.010 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:ddc2cd5d:::45:head 2024-03-26T02:07:49.011 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:2b782207:::46:head 2024-03-26T02:07:49.011 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:7b187fca:::47:head 2024-03-26T02:07:49.011 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:90ecdf6f:::48:head 2024-03-26T02:07:49.011 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:a5ed95fe:::49:head 2024-03-26T02:07:49.011 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:ea0eaa55:::50:head 2024-03-26T02:07:49.011 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:f33ef17b:::51:head 2024-03-26T02:07:49.011 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:a0d1b2f6:::52:head 2024-03-26T02:07:49.011 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:60c5229e:::53:head 2024-03-26T02:07:49.011 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:edcbc575:::54:head 2024-03-26T02:07:49.011 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:102cf253:::55:head 2024-03-26T02:07:49.011 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:efb7fb0b:::56:head 2024-03-26T02:07:49.011 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:50d0a326:::57:head 2024-03-26T02:07:49.011 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:d4dc5daf:::58:head 2024-03-26T02:07:49.011 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:3a130462:::59:head 2024-03-26T02:07:49.011 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:ec87ed71:::60:head 2024-03-26T02:07:49.012 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:d5bc9454:::61:head 2024-03-26T02:07:49.012 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:3ddfe313:::62:head 2024-03-26T02:07:49.012 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:7c2816b9:::63:head 2024-03-26T02:07:49.012 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:47e00e4d:::64:head 2024-03-26T02:07:49.012 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:c6410c18:::65:head 2024-03-26T02:07:49.012 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:b48ed237:::66:head 2024-03-26T02:07:49.012 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:cd63ad31:::67:head 2024-03-26T02:07:49.012 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:b179e92b:::68:head 2024-03-26T02:07:49.012 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:0d9f741a:::69:head 2024-03-26T02:07:49.012 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:6d3352ae:::70:head 2024-03-26T02:07:49.012 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:c6d5c19e:::71:head 2024-03-26T02:07:49.012 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:bc4729c3:::72:head 2024-03-26T02:07:49.012 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:77e930b9:::73:head 2024-03-26T02:07:49.012 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:0abeecfd:::74:head 2024-03-26T02:07:49.013 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:b7c37e15:::75:head 2024-03-26T02:07:49.013 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:b6378398:::76:head 2024-03-26T02:07:49.013 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:02bd68de:::77:head 2024-03-26T02:07:49.013 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:cc795d2d:::78:head 2024-03-26T02:07:49.013 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:630d4fea:::79:head 2024-03-26T02:07:49.013 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:e0d29ef5:::80:head 2024-03-26T02:07:49.013 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:fd6f13d2:::81:head 2024-03-26T02:07:49.013 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:606461d5:::82:head 2024-03-26T02:07:49.013 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:eadbdc43:::83:head 2024-03-26T02:07:49.013 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:8761d0bb:::84:head 2024-03-26T02:07:49.014 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:9ef0186f:::85:head 2024-03-26T02:07:49.014 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:e0d41294:::86:head 2024-03-26T02:07:49.014 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:961de695:::87:head 2024-03-26T02:07:49.014 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:1423148f:::88:head 2024-03-26T02:07:49.014 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:633a8fa2:::89:head 2024-03-26T02:07:49.014 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:a8653809:::90:head 2024-03-26T02:07:49.015 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:3dac8b33:::91:head 2024-03-26T02:07:49.015 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:35aad435:::92:head 2024-03-26T02:07:49.015 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:f6dcc343:::93:head 2024-03-26T02:07:49.015 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:dbbdad87:::94:head 2024-03-26T02:07:49.015 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:1cb48ce0:::95:head 2024-03-26T02:07:49.015 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:03cd461c:::96:head 2024-03-26T02:07:49.015 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:17a4ea99:::97:head 2024-03-26T02:07:49.015 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:9993c9a7:::98:head 2024-03-26T02:07:49.015 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:6394211c:::99:head 2024-03-26T02:07:49.015 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:94c7ae57:::100:head 2024-03-26T02:07:49.016 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:6fdee5bb:::101:head 2024-03-26T02:07:49.016 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:9a477fd1:::102:head 2024-03-26T02:07:49.016 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:eb850916:::103:head 2024-03-26T02:07:49.016 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:affc56b9:::104:head 2024-03-26T02:07:49.016 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:b42dc814:::105:head 2024-03-26T02:07:49.016 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:f319f8f0:::106:head 2024-03-26T02:07:49.016 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:9a40b9de:::107:head 2024-03-26T02:07:49.016 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:8b524f28:::108:head 2024-03-26T02:07:49.016 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:e3de589f:::109:head 2024-03-26T02:07:49.016 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:90f90a5b:::110:head 2024-03-26T02:07:49.016 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:a7b4f1d7:::111:head 2024-03-26T02:07:49.016 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:af51766e:::112:head 2024-03-26T02:07:49.017 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:b6f90bd1:::113:head 2024-03-26T02:07:49.017 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:e0261208:::114:head 2024-03-26T02:07:49.017 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:c9569ef7:::115:head 2024-03-26T02:07:49.017 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:61bebe50:::116:head 2024-03-26T02:07:49.017 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:fe93412b:::117:head 2024-03-26T02:07:49.017 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:d3d38bee:::118:head 2024-03-26T02:07:49.017 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:3100ba0c:::119:head 2024-03-26T02:07:49.017 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:d0560ada:::120:head 2024-03-26T02:07:49.017 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:f0ea8b35:::121:head 2024-03-26T02:07:49.017 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:766f231a:::122:head 2024-03-26T02:07:49.017 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:a07a2582:::123:head 2024-03-26T02:07:49.017 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:bd7c6b3a:::124:head 2024-03-26T02:07:49.017 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:fb2ddaff:::125:head 2024-03-26T02:07:49.017 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:4408e1fe:::126:head 2024-03-26T02:07:49.017 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:ee1df7a7:::127:head 2024-03-26T02:07:49.017 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:c3002909:::128:head 2024-03-26T02:07:49.018 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:4f48ffa9:::129:head 2024-03-26T02:07:49.018 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:edf38733:::130:head 2024-03-26T02:07:49.018 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:c08425c0:::131:head 2024-03-26T02:07:49.018 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:5f902d98:::132:head 2024-03-26T02:07:49.018 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:41ea2c93:::133:head 2024-03-26T02:07:49.018 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:813cee13:::134:head 2024-03-26T02:07:49.018 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:0131818d:::135:head 2024-03-26T02:07:49.018 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:26ba5a85:::136:head 2024-03-26T02:07:49.018 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:381b8a5a:::137:head 2024-03-26T02:07:49.018 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:28797e47:::138:head 2024-03-26T02:07:49.018 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:bfca7f22:::139:head 2024-03-26T02:07:49.018 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:36807075:::140:head 2024-03-26T02:07:49.018 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:80b03975:::141:head 2024-03-26T02:07:49.018 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:5c15709b:::142:head 2024-03-26T02:07:49.019 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:f39ea15e:::143:head 2024-03-26T02:07:49.019 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:ea992956:::144:head 2024-03-26T02:07:49.019 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:48887b1c:::145:head 2024-03-26T02:07:49.019 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:9f24a9dd:::146:head 2024-03-26T02:07:49.019 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:987f100b:::147:head 2024-03-26T02:07:49.019 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:d2dd3581:::148:head 2024-03-26T02:07:49.019 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:7fed1808:::149:head 2024-03-26T02:07:49.019 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:c80b70e9:::150:head 2024-03-26T02:07:49.019 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:85ed90f9:::151:head 2024-03-26T02:07:49.019 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:36428b24:::152:head 2024-03-26T02:07:49.019 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:d044c34a:::153:head 2024-03-26T02:07:49.019 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:7c18bf58:::154:head 2024-03-26T02:07:49.019 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:d1c21232:::155:head 2024-03-26T02:07:49.019 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:a7a3c575:::156:head 2024-03-26T02:07:49.019 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:87da0633:::157:head 2024-03-26T02:07:49.020 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:d5ac3822:::158:head 2024-03-26T02:07:49.020 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:3f20522d:::159:head 2024-03-26T02:07:49.020 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:6ca26563:::160:head 2024-03-26T02:07:49.020 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:532ce135:::161:head 2024-03-26T02:07:49.020 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:c78863e6:::162:head 2024-03-26T02:07:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:49 smithi173 ceph-mon[33839]: pgmap v412: 324 pgs: 324 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:49 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:07:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:49 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi072-82404-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:49 smithi173 ceph-mon[33839]: osdmap e302: 8 total, 8 up, 8 in 2024-03-26T02:07:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:49 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi072-82404-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi072-82404-57"}]: dispatch 2024-03-26T02:07:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:49 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T02:07:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:49 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T02:07:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:49 smithi072 ceph-mon[30177]: pgmap v412: 324 pgs: 324 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:49 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:07:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:49 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi072-82404-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:49 smithi072 ceph-mon[30177]: osdmap e302: 8 total, 8 up, 8 in 2024-03-26T02:07:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:49 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi072-82404-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi072-82404-57"}]: dispatch 2024-03-26T02:07:49.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:49 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T02:07:49.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:49 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T02:07:49.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:49 smithi072 ceph-mon[41619]: pgmap v412: 324 pgs: 324 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:49.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:49 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:07:49.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:49 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi072-82404-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:49.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:49 smithi072 ceph-mon[41619]: osdmap e302: 8 total, 8 up, 8 in 2024-03-26T02:07:49.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:49 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi072-82404-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi072-82404-57"}]: dispatch 2024-03-26T02:07:49.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:49 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T02:07:49.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:49 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T02:07:50.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:50 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-26T02:07:50.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:50 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi072-82692-43","var": "pg_num","format": "json"}]: dispatch 2024-03-26T02:07:50.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:50 smithi173 ceph-mon[33839]: osdmap e303: 8 total, 8 up, 8 in 2024-03-26T02:07:50.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:50 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:50.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:50 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:50.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:50 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-43"}]: dispatch 2024-03-26T02:07:50.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:50 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-43"}]: dispatch 2024-03-26T02:07:50.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:50 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi072-82404-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi072-82404-57"}]': finished 2024-03-26T02:07:50.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:50 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-43"}]': finished 2024-03-26T02:07:50.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:50 smithi173 ceph-mon[33839]: osdmap e304: 8 total, 8 up, 8 in 2024-03-26T02:07:50.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:50 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-26T02:07:50.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:50 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi072-82692-43","var": "pg_num","format": "json"}]: dispatch 2024-03-26T02:07:50.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:50 smithi072 ceph-mon[41619]: osdmap e303: 8 total, 8 up, 8 in 2024-03-26T02:07:50.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:50 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:50.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:50 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:50.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:50 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-43"}]: dispatch 2024-03-26T02:07:50.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:50 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-43"}]: dispatch 2024-03-26T02:07:50.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:50 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi072-82404-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi072-82404-57"}]': finished 2024-03-26T02:07:50.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:50 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-43"}]': finished 2024-03-26T02:07:50.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:50 smithi072 ceph-mon[41619]: osdmap e304: 8 total, 8 up, 8 in 2024-03-26T02:07:50.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:50 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-26T02:07:50.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:50 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi072-82692-43","var": "pg_num","format": "json"}]: dispatch 2024-03-26T02:07:50.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:50 smithi072 ceph-mon[30177]: osdmap e303: 8 total, 8 up, 8 in 2024-03-26T02:07:50.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:50 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:50.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:50 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:07:50.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:50 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-43"}]: dispatch 2024-03-26T02:07:50.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:50 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-43"}]: dispatch 2024-03-26T02:07:50.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:50 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi072-82404-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi072-82404-57"}]': finished 2024-03-26T02:07:50.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:50 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-43"}]': finished 2024-03-26T02:07:50.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:50 smithi072 ceph-mon[30177]: osdmap e304: 8 total, 8 up, 8 in 2024-03-26T02:07:51.057 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:07:50 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:07:51.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:51 smithi173 ceph-mon[33839]: pgmap v415: 324 pgs: 324 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:07:51.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:51 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:51.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:51 smithi173 ceph-mon[33839]: osdmap e305: 8 total, 8 up, 8 in 2024-03-26T02:07:51.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:51 smithi072 ceph-mon[41619]: pgmap v415: 324 pgs: 324 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:07:51.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:51 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:51.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:51 smithi072 ceph-mon[41619]: osdmap e305: 8 total, 8 up, 8 in 2024-03-26T02:07:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:51 smithi072 ceph-mon[30177]: pgmap v415: 324 pgs: 324 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:07:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:51 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:51 smithi072 ceph-mon[30177]: osdmap e305: 8 total, 8 up, 8 in 2024-03-26T02:07:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:52 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:07:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:07:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:52 smithi173 ceph-mon[33839]: osdmap e306: 8 total, 8 up, 8 in 2024-03-26T02:07:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:52 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:52 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi072-82404-57"}]: dispatch 2024-03-26T02:07:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:52 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:52.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:52 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:07:52.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:07:52.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:52 smithi072 ceph-mon[30177]: osdmap e306: 8 total, 8 up, 8 in 2024-03-26T02:07:52.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:52 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:52.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:52 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi072-82404-57"}]: dispatch 2024-03-26T02:07:52.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:52 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:52 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:07:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:07:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:52 smithi072 ceph-mon[41619]: osdmap e306: 8 total, 8 up, 8 in 2024-03-26T02:07:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:52 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:52 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi072-82404-57"}]: dispatch 2024-03-26T02:07:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:52 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:07:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:53 smithi072 ceph-mon[30177]: pgmap v418: 300 pgs: 8 creating+peering, 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:53 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi072-82404-57"}]': finished 2024-03-26T02:07:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:53 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:07:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:53 smithi072 ceph-mon[30177]: osdmap e307: 8 total, 8 up, 8 in 2024-03-26T02:07:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:53 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi072-82404-57"}]: dispatch 2024-03-26T02:07:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:53 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:53 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:53 smithi072 ceph-mon[41619]: pgmap v418: 300 pgs: 8 creating+peering, 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:53 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi072-82404-57"}]': finished 2024-03-26T02:07:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:53 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:07:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:53 smithi072 ceph-mon[41619]: osdmap e307: 8 total, 8 up, 8 in 2024-03-26T02:07:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:53 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi072-82404-57"}]: dispatch 2024-03-26T02:07:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:53 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:53 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:53 smithi173 ceph-mon[33839]: pgmap v418: 300 pgs: 8 creating+peering, 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:53 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi072-82404-57"}]': finished 2024-03-26T02:07:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:53 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:07:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:53 smithi173 ceph-mon[33839]: osdmap e307: 8 total, 8 up, 8 in 2024-03-26T02:07:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:53 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi072-82404-57"}]: dispatch 2024-03-26T02:07:53.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:53 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:53.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:53 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:07:55.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:54 smithi173 ceph-mon[33839]: pgmap v421: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:55.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:54 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi072-82404-57"}]': finished 2024-03-26T02:07:55.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:54 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:07:55.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:54 smithi173 ceph-mon[33839]: osdmap e308: 8 total, 8 up, 8 in 2024-03-26T02:07:55.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:54 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T02:07:55.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:54 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T02:07:55.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:54 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi072-82404-58"}]: dispatch 2024-03-26T02:07:55.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:54 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi072-82404-58"}]: dispatch 2024-03-26T02:07:55.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:54 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi072-82404-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:55.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:07:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:07:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:07:55.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:54 smithi072 ceph-mon[30177]: pgmap v421: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:55.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:54 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi072-82404-57"}]': finished 2024-03-26T02:07:55.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:54 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:07:55.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:54 smithi072 ceph-mon[30177]: osdmap e308: 8 total, 8 up, 8 in 2024-03-26T02:07:55.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:54 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T02:07:55.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:54 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T02:07:55.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:54 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi072-82404-58"}]: dispatch 2024-03-26T02:07:55.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:54 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi072-82404-58"}]: dispatch 2024-03-26T02:07:55.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:54 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi072-82404-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:55.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:54 smithi072 ceph-mon[41619]: pgmap v421: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:55.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:54 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2278343958' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi072-82404-57"}]': finished 2024-03-26T02:07:55.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:54 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:07:55.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:54 smithi072 ceph-mon[41619]: osdmap e308: 8 total, 8 up, 8 in 2024-03-26T02:07:55.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:54 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T02:07:55.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:54 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T02:07:55.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:54 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi072-82404-58"}]: dispatch 2024-03-26T02:07:55.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:54 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi072-82404-58"}]: dispatch 2024-03-26T02:07:55.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:54 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi072-82404-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:07:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:56 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_count","val": "3"}]': finished 2024-03-26T02:07:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:56 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi072-82404-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:56 smithi173 ceph-mon[33839]: osdmap e309: 8 total, 8 up, 8 in 2024-03-26T02:07:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:56 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T02:07:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:56 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi072-82404-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi072-82404-58"}]: dispatch 2024-03-26T02:07:56.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:56 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T02:07:56.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:56 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_period","val": "3"}]': finished 2024-03-26T02:07:56.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:56 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_count","val": "3"}]': finished 2024-03-26T02:07:56.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:56 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi072-82404-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:56.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:56 smithi072 ceph-mon[30177]: osdmap e309: 8 total, 8 up, 8 in 2024-03-26T02:07:56.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:56 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T02:07:56.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:56 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi072-82404-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi072-82404-58"}]: dispatch 2024-03-26T02:07:56.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:56 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T02:07:56.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:56 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_period","val": "3"}]': finished 2024-03-26T02:07:56.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:56 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_count","val": "3"}]': finished 2024-03-26T02:07:56.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:56 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi072-82404-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:07:56.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:56 smithi072 ceph-mon[41619]: osdmap e309: 8 total, 8 up, 8 in 2024-03-26T02:07:56.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:56 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T02:07:56.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:56 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi072-82404-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi072-82404-58"}]: dispatch 2024-03-26T02:07:56.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:56 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T02:07:56.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:56 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_period","val": "3"}]': finished 2024-03-26T02:07:57.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:57 smithi173 ceph-mon[33839]: pgmap v424: 324 pgs: 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:57.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:57 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:07:57.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:57 smithi173 ceph-mon[33839]: osdmap e310: 8 total, 8 up, 8 in 2024-03-26T02:07:57.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:57 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:07:57.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:57 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi072-82404-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi072-82404-58"}]': finished 2024-03-26T02:07:57.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:57 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T02:07:57.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:57 smithi173 ceph-mon[33839]: osdmap e311: 8 total, 8 up, 8 in 2024-03-26T02:07:57.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:57 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T02:07:57.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:57 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T02:07:57.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:57 smithi072 ceph-mon[30177]: pgmap v424: 324 pgs: 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:57.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:57 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:07:57.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:57 smithi072 ceph-mon[30177]: osdmap e310: 8 total, 8 up, 8 in 2024-03-26T02:07:57.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:57 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:07:57.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:57 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi072-82404-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi072-82404-58"}]': finished 2024-03-26T02:07:57.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:57 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T02:07:57.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:57 smithi072 ceph-mon[30177]: osdmap e311: 8 total, 8 up, 8 in 2024-03-26T02:07:57.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:57 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T02:07:57.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:57 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T02:07:57.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:57 smithi072 ceph-mon[41619]: pgmap v424: 324 pgs: 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:57.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:57 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:07:57.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:57 smithi072 ceph-mon[41619]: osdmap e310: 8 total, 8 up, 8 in 2024-03-26T02:07:57.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:57 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:07:57.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:57 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi072-82404-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi072-82404-58"}]': finished 2024-03-26T02:07:57.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:57 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T02:07:57.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:57 smithi072 ceph-mon[41619]: osdmap e311: 8 total, 8 up, 8 in 2024-03-26T02:07:57.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:57 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T02:07:57.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:57 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T02:07:59.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:58 smithi173 ceph-mon[33839]: pgmap v427: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:59.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:58 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T02:07:59.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:58 smithi173 ceph-mon[33839]: osdmap e312: 8 total, 8 up, 8 in 2024-03-26T02:07:59.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:58 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:59.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:58 smithi072 ceph-mon[30177]: pgmap v427: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:59.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:58 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T02:07:59.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:58 smithi072 ceph-mon[30177]: osdmap e312: 8 total, 8 up, 8 in 2024-03-26T02:07:59.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:58 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:07:59.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:58 smithi072 ceph-mon[41619]: pgmap v427: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:07:59.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:58 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T02:07:59.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:58 smithi072 ceph-mon[41619]: osdmap e312: 8 total, 8 up, 8 in 2024-03-26T02:07:59.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:58 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:00.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:59 smithi173 ceph-mon[33839]: osdmap e313: 8 total, 8 up, 8 in 2024-03-26T02:08:00.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:59 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi072-82404-58"}]: dispatch 2024-03-26T02:08:00.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:59 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi072-82404-58"}]': finished 2024-03-26T02:08:00.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:07:59 smithi173 ceph-mon[33839]: osdmap e314: 8 total, 8 up, 8 in 2024-03-26T02:08:00.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:59 smithi072 ceph-mon[30177]: osdmap e313: 8 total, 8 up, 8 in 2024-03-26T02:08:00.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:59 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi072-82404-58"}]: dispatch 2024-03-26T02:08:00.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:59 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi072-82404-58"}]': finished 2024-03-26T02:08:00.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:07:59 smithi072 ceph-mon[30177]: osdmap e314: 8 total, 8 up, 8 in 2024-03-26T02:08:00.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:59 smithi072 ceph-mon[41619]: osdmap e313: 8 total, 8 up, 8 in 2024-03-26T02:08:00.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:59 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi072-82404-58"}]: dispatch 2024-03-26T02:08:00.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:59 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi072-82404-58"}]': finished 2024-03-26T02:08:00.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:07:59 smithi072 ceph-mon[41619]: osdmap e314: 8 total, 8 up, 8 in 2024-03-26T02:08:01.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:08:00 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:08:01.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:00 smithi173 ceph-mon[33839]: pgmap v430: 324 pgs: 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:08:01.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi072-82404-58"}]: dispatch 2024-03-26T02:08:01.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi072-82404-58"}]': finished 2024-03-26T02:08:01.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:00 smithi173 ceph-mon[33839]: osdmap e315: 8 total, 8 up, 8 in 2024-03-26T02:08:01.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:00 smithi072 ceph-mon[30177]: pgmap v430: 324 pgs: 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:08:01.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi072-82404-58"}]: dispatch 2024-03-26T02:08:01.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi072-82404-58"}]': finished 2024-03-26T02:08:01.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:00 smithi072 ceph-mon[30177]: osdmap e315: 8 total, 8 up, 8 in 2024-03-26T02:08:01.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:00 smithi072 ceph-mon[41619]: pgmap v430: 324 pgs: 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:08:01.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi072-82404-58"}]: dispatch 2024-03-26T02:08:01.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3825275009' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi072-82404-58"}]': finished 2024-03-26T02:08:01.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:00 smithi072 ceph-mon[41619]: osdmap e315: 8 total, 8 up, 8 in 2024-03-26T02:08:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:01 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:08:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi072-82404-59"}]: dispatch 2024-03-26T02:08:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi072-82404-59"}]: dispatch 2024-03-26T02:08:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi072-82404-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:01 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:08:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi072-82404-59"}]: dispatch 2024-03-26T02:08:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi072-82404-59"}]: dispatch 2024-03-26T02:08:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi072-82404-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:01 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:08:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi072-82404-59"}]: dispatch 2024-03-26T02:08:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi072-82404-59"}]: dispatch 2024-03-26T02:08:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi072-82404-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:03 smithi072 ceph-mon[30177]: pgmap v433: 324 pgs: 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-26T02:08:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi072-82404-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:08:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:03 smithi072 ceph-mon[30177]: osdmap e316: 8 total, 8 up, 8 in 2024-03-26T02:08:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi072-82404-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi072-82404-59"}]: dispatch 2024-03-26T02:08:03.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:03 smithi072 ceph-mon[41619]: pgmap v433: 324 pgs: 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-26T02:08:03.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi072-82404-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:08:03.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:03 smithi072 ceph-mon[41619]: osdmap e316: 8 total, 8 up, 8 in 2024-03-26T02:08:03.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi072-82404-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi072-82404-59"}]: dispatch 2024-03-26T02:08:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:03 smithi173 ceph-mon[33839]: pgmap v433: 324 pgs: 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-26T02:08:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi072-82404-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:08:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:03 smithi173 ceph-mon[33839]: osdmap e316: 8 total, 8 up, 8 in 2024-03-26T02:08:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi072-82404-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi072-82404-59"}]: dispatch 2024-03-26T02:08:04.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:04 smithi072 ceph-mon[30177]: osdmap e317: 8 total, 8 up, 8 in 2024-03-26T02:08:04.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:04 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi072-82404-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi072-82404-59"}]': finished 2024-03-26T02:08:04.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:04 smithi072 ceph-mon[30177]: osdmap e318: 8 total, 8 up, 8 in 2024-03-26T02:08:04.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:04 smithi072 ceph-mon[41619]: osdmap e317: 8 total, 8 up, 8 in 2024-03-26T02:08:04.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:04 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi072-82404-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi072-82404-59"}]': finished 2024-03-26T02:08:04.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:04 smithi072 ceph-mon[41619]: osdmap e318: 8 total, 8 up, 8 in 2024-03-26T02:08:04.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:04 smithi173 ceph-mon[33839]: osdmap e317: 8 total, 8 up, 8 in 2024-03-26T02:08:04.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:04 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi072-82404-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi072-82404-59"}]': finished 2024-03-26T02:08:04.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:04 smithi173 ceph-mon[33839]: osdmap e318: 8 total, 8 up, 8 in 2024-03-26T02:08:05.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:05 smithi072 ceph-mon[30177]: pgmap v436: 324 pgs: 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-26T02:08:05.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:05 smithi072 ceph-mon[30177]: osdmap e319: 8 total, 8 up, 8 in 2024-03-26T02:08:05.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:08:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:08:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:08:05.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:05 smithi072 ceph-mon[41619]: pgmap v436: 324 pgs: 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-26T02:08:05.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:05 smithi072 ceph-mon[41619]: osdmap e319: 8 total, 8 up, 8 in 2024-03-26T02:08:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:05 smithi173 ceph-mon[33839]: pgmap v436: 324 pgs: 324 active+clean; 8.4 MiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 1 op/s 2024-03-26T02:08:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:05 smithi173 ceph-mon[33839]: osdmap e319: 8 total, 8 up, 8 in 2024-03-26T02:08:06.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:06 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:06.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:06 smithi072 ceph-mon[30177]: osdmap e320: 8 total, 8 up, 8 in 2024-03-26T02:08:06.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi072-82404-59"}]: dispatch 2024-03-26T02:08:06.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:06 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:06.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:06 smithi072 ceph-mon[41619]: osdmap e320: 8 total, 8 up, 8 in 2024-03-26T02:08:06.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi072-82404-59"}]: dispatch 2024-03-26T02:08:06.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:06 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:06.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:06 smithi173 ceph-mon[33839]: osdmap e320: 8 total, 8 up, 8 in 2024-03-26T02:08:06.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi072-82404-59"}]: dispatch 2024-03-26T02:08:07.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:07 smithi072 ceph-mon[30177]: pgmap v439: 332 pgs: 332 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-03-26T02:08:07.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:07 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:08:07.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi072-82404-59"}]': finished 2024-03-26T02:08:07.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:07 smithi072 ceph-mon[30177]: osdmap e321: 8 total, 8 up, 8 in 2024-03-26T02:08:07.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi072-82404-59"}]: dispatch 2024-03-26T02:08:07.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:07 smithi072 ceph-mon[41619]: pgmap v439: 332 pgs: 332 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-03-26T02:08:07.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:07 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:08:07.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi072-82404-59"}]': finished 2024-03-26T02:08:07.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:07 smithi072 ceph-mon[41619]: osdmap e321: 8 total, 8 up, 8 in 2024-03-26T02:08:07.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi072-82404-59"}]: dispatch 2024-03-26T02:08:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:07 smithi173 ceph-mon[33839]: pgmap v439: 332 pgs: 332 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-03-26T02:08:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:07 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:08:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi072-82404-59"}]': finished 2024-03-26T02:08:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:07 smithi173 ceph-mon[33839]: osdmap e321: 8 total, 8 up, 8 in 2024-03-26T02:08:07.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi072-82404-59"}]: dispatch 2024-03-26T02:08:09.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:08 smithi072 ceph-mon[41619]: pgmap v442: 324 pgs: 324 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-26T02:08:09.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi072-82404-59"}]': finished 2024-03-26T02:08:09.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:08 smithi072 ceph-mon[41619]: osdmap e322: 8 total, 8 up, 8 in 2024-03-26T02:08:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:08 smithi072 ceph-mon[30177]: pgmap v442: 324 pgs: 324 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-26T02:08:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi072-82404-59"}]': finished 2024-03-26T02:08:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:08 smithi072 ceph-mon[30177]: osdmap e322: 8 total, 8 up, 8 in 2024-03-26T02:08:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:08 smithi173 ceph-mon[33839]: pgmap v442: 324 pgs: 324 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-26T02:08:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4184310698' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi072-82404-59"}]': finished 2024-03-26T02:08:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:08 smithi173 ceph-mon[33839]: osdmap e322: 8 total, 8 up, 8 in 2024-03-26T02:08:10.121 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:09 smithi072 ceph-mon[41619]: osdmap e323: 8 total, 8 up, 8 in 2024-03-26T02:08:10.121 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3985579994' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi072-82404-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:08:10.121 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi072-82404-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:08:10.121 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:09 smithi072 ceph-mon[30177]: osdmap e323: 8 total, 8 up, 8 in 2024-03-26T02:08:10.121 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3985579994' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi072-82404-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:08:10.122 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi072-82404-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:08:10.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:09 smithi173 ceph-mon[33839]: osdmap e323: 8 total, 8 up, 8 in 2024-03-26T02:08:10.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3985579994' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi072-82404-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:08:10.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi072-82404-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:08:10.855 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:7462ddf6:::o.RoundTripAppendPP (3008 ms) 2024-03-26T02:08:10.855 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-03-26T02:08:10.855 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3201 ms) 2024-03-26T02:08:10.855 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-03-26T02:08:10.855 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (2758 ms) 2024-03-26T02:08:10.855 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-03-26T02:08:10.855 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3015 ms) 2024-03-26T02:08:10.855 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-03-26T02:08:10.855 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: setting pool EIO 2024-03-26T02:08:10.855 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: max_success 98, min_failed 99 2024-03-26T02:08:10.856 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4007 ms) 2024-03-26T02:08:10.856 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-03-26T02:08:10.856 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3009 ms) 2024-03-26T02:08:10.856 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (131618 ms total) 2024-03-26T02:08:10.856 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: 2024-03-26T02:08:10.856 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-03-26T02:08:10.856 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-03-26T02:08:10.856 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3001 ms) 2024-03-26T02:08:10.856 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-03-26T02:08:10.856 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9022 ms) 2024-03-26T02:08:10.856 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-03-26T02:08:10.856 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (14994 ms) 2024-03-26T02:08:10.856 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-03-26T02:08:10.856 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3018 ms) 2024-03-26T02:08:10.856 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (30035 ms total) 2024-03-26T02:08:10.856 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: 2024-03-26T02:08:10.857 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-03-26T02:08:10.857 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-26T02:08:10.857 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3017 ms) 2024-03-26T02:08:10.857 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3017 ms total) 2024-03-26T02:08:10.857 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: 2024-03-26T02:08:10.857 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-03-26T02:08:10.857 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-03-26T02:08:10.857 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13936 ms) 2024-03-26T02:08:10.857 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-03-26T02:08:10.857 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7000 ms) 2024-03-26T02:08:10.857 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-03-26T02:08:10.857 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7027 ms) 2024-03-26T02:08:10.857 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-03-26T02:08:10.857 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7048 ms) 2024-03-26T02:08:10.857 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-03-26T02:08:10.858 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3237 ms) 2024-03-26T02:08:10.858 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-03-26T02:08:10.858 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6803 ms) 2024-03-26T02:08:10.858 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-03-26T02:08:10.858 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (7131 ms) 2024-03-26T02:08:10.858 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-03-26T02:08:10.858 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (6780 ms) 2024-03-26T02:08:10.858 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-03-26T02:08:10.858 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7030 ms) 2024-03-26T02:08:10.858 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-03-26T02:08:10.858 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7067 ms) 2024-03-26T02:08:10.858 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-03-26T02:08:10.858 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7020 ms) 2024-03-26T02:08:10.858 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-03-26T02:08:10.859 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7015 ms) 2024-03-26T02:08:10.859 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-03-26T02:08:10.859 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7025 ms) 2024-03-26T02:08:10.859 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-03-26T02:08:11.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:08:10 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:08:11.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:10 smithi173 ceph-mon[33839]: pgmap v445: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:08:11.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi072-82404-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:08:11.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:10 smithi173 ceph-mon[33839]: osdmap e324: 8 total, 8 up, 8 in 2024-03-26T02:08:11.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:11.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:11.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-45"}]: dispatch 2024-03-26T02:08:11.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-45"}]: dispatch 2024-03-26T02:08:11.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:10 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:11.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:10 smithi072 ceph-mon[41619]: pgmap v445: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:08:11.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi072-82404-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:08:11.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:10 smithi072 ceph-mon[41619]: osdmap e324: 8 total, 8 up, 8 in 2024-03-26T02:08:11.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:11.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-45"}]: dispatch 2024-03-26T02:08:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-45"}]: dispatch 2024-03-26T02:08:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:10 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:10 smithi072 ceph-mon[30177]: pgmap v445: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:08:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi072-82404-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:08:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:10 smithi072 ceph-mon[30177]: osdmap e324: 8 total, 8 up, 8 in 2024-03-26T02:08:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-45"}]: dispatch 2024-03-26T02:08:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-45"}]: dispatch 2024-03-26T02:08:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:10 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:12.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:11 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:08:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:11 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-45"}]': finished 2024-03-26T02:08:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:11 smithi072 ceph-mon[41619]: osdmap e325: 8 total, 8 up, 8 in 2024-03-26T02:08:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:11 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1408574197' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:11 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:11 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1408574197' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:11 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:11 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1408574197' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi072-82404-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:11 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi072-82404-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:12.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:11 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:08:12.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:11 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-45"}]': finished 2024-03-26T02:08:12.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:11 smithi072 ceph-mon[30177]: osdmap e325: 8 total, 8 up, 8 in 2024-03-26T02:08:12.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:11 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1408574197' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:12.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:11 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:12.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:11 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1408574197' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:12.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:11 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:12.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:11 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1408574197' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi072-82404-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:12.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:11 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi072-82404-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:11 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:08:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:11 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-45"}]': finished 2024-03-26T02:08:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:11 smithi173 ceph-mon[33839]: osdmap e325: 8 total, 8 up, 8 in 2024-03-26T02:08:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:11 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1408574197' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:11 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:12.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:11 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1408574197' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:12.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:11 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:12.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:11 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1408574197' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi072-82404-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:12.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:11 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi072-82404-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:13.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:13 smithi072 ceph-mon[30177]: pgmap v448: 324 pgs: 324 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-26T02:08:13.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:13 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi072-82404-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:08:13.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:13 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1408574197' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi072-82404-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:13.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:13 smithi072 ceph-mon[30177]: osdmap e326: 8 total, 8 up, 8 in 2024-03-26T02:08:13.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:13 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi072-82404-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:13.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:13 smithi072 ceph-mon[41619]: pgmap v448: 324 pgs: 324 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-26T02:08:13.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:13 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi072-82404-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:08:13.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:13 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1408574197' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi072-82404-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:13.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:13 smithi072 ceph-mon[41619]: osdmap e326: 8 total, 8 up, 8 in 2024-03-26T02:08:13.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:13 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi072-82404-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:13 smithi173 ceph-mon[33839]: pgmap v448: 324 pgs: 324 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-26T02:08:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:13 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi072-82404-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:08:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:13 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1408574197' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi072-82404-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:13 smithi173 ceph-mon[33839]: osdmap e326: 8 total, 8 up, 8 in 2024-03-26T02:08:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:13 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi072-82404-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:14.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:14 smithi072 ceph-mon[30177]: osdmap e327: 8 total, 8 up, 8 in 2024-03-26T02:08:14.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:14 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:08:14.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:14 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:08:14.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:14 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi072-82404-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi072-82404-61"}]': finished 2024-03-26T02:08:14.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:14 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:08:14.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:14 smithi072 ceph-mon[30177]: osdmap e328: 8 total, 8 up, 8 in 2024-03-26T02:08:14.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:14 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:08:14.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:14 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:08:14.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:14 smithi072 ceph-mon[41619]: osdmap e327: 8 total, 8 up, 8 in 2024-03-26T02:08:14.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:14 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:08:14.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:14 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:08:14.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:14 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi072-82404-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi072-82404-61"}]': finished 2024-03-26T02:08:14.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:14 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:08:14.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:14 smithi072 ceph-mon[41619]: osdmap e328: 8 total, 8 up, 8 in 2024-03-26T02:08:14.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:14 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:08:14.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:14 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:08:14.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:14 smithi173 ceph-mon[33839]: osdmap e327: 8 total, 8 up, 8 in 2024-03-26T02:08:14.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:14 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:08:14.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:14 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:08:14.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:14 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi072-82404-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi072-82404-61"}]': finished 2024-03-26T02:08:14.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:14 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:08:14.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:14 smithi173 ceph-mon[33839]: osdmap e328: 8 total, 8 up, 8 in 2024-03-26T02:08:14.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:14 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:08:14.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:14 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:08:15.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:08:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:08:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:08:15.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:15 smithi072 ceph-mon[30177]: pgmap v451: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:08:15.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:08:15.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:15 smithi072 ceph-mon[30177]: osdmap e329: 8 total, 8 up, 8 in 2024-03-26T02:08:15.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-47"}]: dispatch 2024-03-26T02:08:15.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-47"}]: dispatch 2024-03-26T02:08:15.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:15 smithi072 ceph-mon[41619]: pgmap v451: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:08:15.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:08:15.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:15 smithi072 ceph-mon[41619]: osdmap e329: 8 total, 8 up, 8 in 2024-03-26T02:08:15.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-47"}]: dispatch 2024-03-26T02:08:15.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-47"}]: dispatch 2024-03-26T02:08:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:15 smithi173 ceph-mon[33839]: pgmap v451: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:08:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:08:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:15 smithi173 ceph-mon[33839]: osdmap e329: 8 total, 8 up, 8 in 2024-03-26T02:08:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-47"}]: dispatch 2024-03-26T02:08:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-47"}]: dispatch 2024-03-26T02:08:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:16 smithi072 ceph-mon[30177]: pgmap v454: 332 pgs: 332 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T02:08:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:16 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-47"}]': finished 2024-03-26T02:08:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:16 smithi072 ceph-mon[30177]: osdmap e330: 8 total, 8 up, 8 in 2024-03-26T02:08:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:16 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1408574197' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:17.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:16 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-47", "mode": "writeback"}]: dispatch 2024-03-26T02:08:17.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:16 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:17.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:16 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-47", "mode": "writeback"}]: dispatch 2024-03-26T02:08:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:16 smithi072 ceph-mon[41619]: pgmap v454: 332 pgs: 332 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T02:08:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:16 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-47"}]': finished 2024-03-26T02:08:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:16 smithi072 ceph-mon[41619]: osdmap e330: 8 total, 8 up, 8 in 2024-03-26T02:08:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:16 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1408574197' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:16 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-47", "mode": "writeback"}]: dispatch 2024-03-26T02:08:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:16 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:16 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-47", "mode": "writeback"}]: dispatch 2024-03-26T02:08:17.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:16 smithi173 ceph-mon[33839]: pgmap v454: 332 pgs: 332 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T02:08:17.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:16 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-47"}]': finished 2024-03-26T02:08:17.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:16 smithi173 ceph-mon[33839]: osdmap e330: 8 total, 8 up, 8 in 2024-03-26T02:08:17.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:16 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1408574197' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:17.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:16 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-47", "mode": "writeback"}]: dispatch 2024-03-26T02:08:17.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:16 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:17.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:16 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-47", "mode": "writeback"}]: dispatch 2024-03-26T02:08:18.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:17 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:08:18.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:17 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi072-82404-61"}]': finished 2024-03-26T02:08:18.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:17 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-47", "mode": "writeback"}]': finished 2024-03-26T02:08:18.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:17 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1408574197' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:18.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:17 smithi072 ceph-mon[30177]: osdmap e331: 8 total, 8 up, 8 in 2024-03-26T02:08:18.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:17 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:18.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:17 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:08:18.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:17 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:08:18.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:17 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi072-82404-61"}]': finished 2024-03-26T02:08:18.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:17 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_count","val": "2"}]': finished 2024-03-26T02:08:18.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:17 smithi072 ceph-mon[30177]: osdmap e332: 8 total, 8 up, 8 in 2024-03-26T02:08:18.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:17 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:08:18.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:17 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:08:18.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:17 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:08:18.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:17 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi072-82404-61"}]': finished 2024-03-26T02:08:18.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:17 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-47", "mode": "writeback"}]': finished 2024-03-26T02:08:18.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:17 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1408574197' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:18.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:17 smithi072 ceph-mon[41619]: osdmap e331: 8 total, 8 up, 8 in 2024-03-26T02:08:18.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:17 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:18.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:17 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:08:18.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:17 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:08:18.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:17 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi072-82404-61"}]': finished 2024-03-26T02:08:18.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:17 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_count","val": "2"}]': finished 2024-03-26T02:08:18.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:17 smithi072 ceph-mon[41619]: osdmap e332: 8 total, 8 up, 8 in 2024-03-26T02:08:18.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:17 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:08:18.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:17 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:08:18.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:17 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:08:18.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:17 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi072-82404-61"}]': finished 2024-03-26T02:08:18.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:17 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-47", "mode": "writeback"}]': finished 2024-03-26T02:08:18.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:17 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1408574197' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:18.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:17 smithi173 ceph-mon[33839]: osdmap e331: 8 total, 8 up, 8 in 2024-03-26T02:08:18.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:17 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi072-82404-61"}]: dispatch 2024-03-26T02:08:18.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:17 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:08:18.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:17 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:08:18.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:17 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi072-82404-61"}]': finished 2024-03-26T02:08:18.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:17 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_count","val": "2"}]': finished 2024-03-26T02:08:18.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:17 smithi173 ceph-mon[33839]: osdmap e332: 8 total, 8 up, 8 in 2024-03-26T02:08:18.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:17 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:08:18.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:17 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:08:19.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[30177]: pgmap v457: 324 pgs: 324 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T02:08:19.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:19.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4278720799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:19.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:19.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4278720799' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:19.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:19.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4278720799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi072-82404-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:19.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi072-82404-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:19.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:08:19.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi072-82404-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:08:19.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[30177]: osdmap e333: 8 total, 8 up, 8 in 2024-03-26T02:08:19.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:08:19.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4278720799' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi072-82404-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:19.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:08:19.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi072-82404-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:19.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[41619]: pgmap v457: 324 pgs: 324 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T02:08:19.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:19.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4278720799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:19.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:19.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4278720799' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:19.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:19.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4278720799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi072-82404-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:19.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi072-82404-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:19.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:08:19.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi072-82404-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:08:19.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[41619]: osdmap e333: 8 total, 8 up, 8 in 2024-03-26T02:08:19.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:08:19.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4278720799' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi072-82404-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:19.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:08:19.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi072-82404-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:19.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:18 smithi173 ceph-mon[33839]: pgmap v457: 324 pgs: 324 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T02:08:19.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:18 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:19.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4278720799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:19.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:19.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4278720799' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:19.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:19.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4278720799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi072-82404-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:19.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi072-82404-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:19.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:08:19.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi072-82404-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:08:19.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:18 smithi173 ceph-mon[33839]: osdmap e333: 8 total, 8 up, 8 in 2024-03-26T02:08:19.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:08:19.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4278720799' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi072-82404-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:19.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:08:19.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi072-82404-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:20.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:20 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:08:20.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:20 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T02:08:20.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:20 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:08:20.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:20 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T02:08:20.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:20 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:08:20.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:20 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T02:08:21.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:08:20 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:08:21.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:21 smithi072 ceph-mon[30177]: pgmap v460: 324 pgs: 324 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:08:21.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T02:08:21.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:21 smithi072 ceph-mon[30177]: osdmap e334: 8 total, 8 up, 8 in 2024-03-26T02:08:21.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:21 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T02:08:21.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:21 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi072-82404-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi072-82404-62"}]': finished 2024-03-26T02:08:21.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:21 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T02:08:21.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:21 smithi072 ceph-mon[30177]: osdmap e335: 8 total, 8 up, 8 in 2024-03-26T02:08:21.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T02:08:21.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:21 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T02:08:21.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:21 smithi072 ceph-mon[41619]: pgmap v460: 324 pgs: 324 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:08:21.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T02:08:21.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:21 smithi072 ceph-mon[41619]: osdmap e334: 8 total, 8 up, 8 in 2024-03-26T02:08:21.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:21 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T02:08:21.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:21 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi072-82404-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi072-82404-62"}]': finished 2024-03-26T02:08:21.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:21 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T02:08:21.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:21 smithi072 ceph-mon[41619]: osdmap e335: 8 total, 8 up, 8 in 2024-03-26T02:08:21.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T02:08:21.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:21 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T02:08:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:21 smithi173 ceph-mon[33839]: pgmap v460: 324 pgs: 324 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:08:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T02:08:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:21 smithi173 ceph-mon[33839]: osdmap e334: 8 total, 8 up, 8 in 2024-03-26T02:08:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:21 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T02:08:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:21 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi072-82404-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi072-82404-62"}]': finished 2024-03-26T02:08:21.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:21 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T02:08:21.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:21 smithi173 ceph-mon[33839]: osdmap e335: 8 total, 8 up, 8 in 2024-03-26T02:08:21.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T02:08:21.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:21 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T02:08:22.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:22 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:08:22.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:22 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:08:22.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:22 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T02:08:22.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:22 smithi072 ceph-mon[30177]: osdmap e336: 8 total, 8 up, 8 in 2024-03-26T02:08:22.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:22 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T02:08:22.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:22 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T02:08:22.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:22 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:08:22.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:22 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:08:22.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:22 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T02:08:22.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:22 smithi072 ceph-mon[41619]: osdmap e336: 8 total, 8 up, 8 in 2024-03-26T02:08:22.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:22 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T02:08:22.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:22 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T02:08:22.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:22 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:08:22.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:22 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:08:22.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:22 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T02:08:22.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:22 smithi173 ceph-mon[33839]: osdmap e336: 8 total, 8 up, 8 in 2024-03-26T02:08:22.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:22 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T02:08:22.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:22 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T02:08:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:23 smithi072 ceph-mon[30177]: pgmap v463: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:08:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:23 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T02:08:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:23 smithi072 ceph-mon[30177]: osdmap e337: 8 total, 8 up, 8 in 2024-03-26T02:08:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:23 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4278720799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:23.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:23 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:23.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:23 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:23.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:23 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:23 smithi072 ceph-mon[41619]: pgmap v463: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:08:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:23 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T02:08:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:23 smithi072 ceph-mon[41619]: osdmap e337: 8 total, 8 up, 8 in 2024-03-26T02:08:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:23 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4278720799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:23 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:23 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:23 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:23 smithi173 ceph-mon[33839]: pgmap v463: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:08:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:23 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T02:08:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:23 smithi173 ceph-mon[33839]: osdmap e337: 8 total, 8 up, 8 in 2024-03-26T02:08:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:23 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4278720799' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:23 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:23 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:23 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:25.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:08:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:08:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:08:25.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:24 smithi072 ceph-mon[41619]: pgmap v466: 324 pgs: 324 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:08:25.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:24 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi072-82404-62"}]': finished 2024-03-26T02:08:25.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:24 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:08:25.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:24 smithi072 ceph-mon[41619]: osdmap e338: 8 total, 8 up, 8 in 2024-03-26T02:08:25.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:24 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-47"}]: dispatch 2024-03-26T02:08:25.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:24 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/4278720799' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:25.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:24 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-47"}]: dispatch 2024-03-26T02:08:25.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:24 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:25.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:24 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:25.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:24 smithi072 ceph-mon[30177]: pgmap v466: 324 pgs: 324 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:08:25.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:24 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi072-82404-62"}]': finished 2024-03-26T02:08:25.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:24 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:08:25.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:24 smithi072 ceph-mon[30177]: osdmap e338: 8 total, 8 up, 8 in 2024-03-26T02:08:25.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:24 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-47"}]: dispatch 2024-03-26T02:08:25.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:24 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/4278720799' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:25.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:24 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-47"}]: dispatch 2024-03-26T02:08:25.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:24 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:25.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:24 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:24 smithi173 ceph-mon[33839]: pgmap v466: 324 pgs: 324 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:08:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:24 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi072-82404-62"}]': finished 2024-03-26T02:08:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:24 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:08:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:24 smithi173 ceph-mon[33839]: osdmap e338: 8 total, 8 up, 8 in 2024-03-26T02:08:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:24 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-47"}]: dispatch 2024-03-26T02:08:25.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:24 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/4278720799' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:25.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:24 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-47"}]: dispatch 2024-03-26T02:08:25.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:24 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi072-82404-62"}]: dispatch 2024-03-26T02:08:25.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:24 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:26.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:25 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-47"}]': finished 2024-03-26T02:08:26.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:25 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi072-82404-62"}]': finished 2024-03-26T02:08:26.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:25 smithi072 ceph-mon[30177]: osdmap e339: 8 total, 8 up, 8 in 2024-03-26T02:08:26.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1980931521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:26.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:25 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:26.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1980931521' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:26.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:25 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:26.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1980931521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi072-82404-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:26.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:25 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi072-82404-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:26.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:26.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:25 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:26.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-47"}]: dispatch 2024-03-26T02:08:26.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:25 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-47"}]: dispatch 2024-03-26T02:08:26.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:25 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-47"}]': finished 2024-03-26T02:08:26.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:25 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi072-82404-62"}]': finished 2024-03-26T02:08:26.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:25 smithi072 ceph-mon[41619]: osdmap e339: 8 total, 8 up, 8 in 2024-03-26T02:08:26.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1980931521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:26.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:25 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:26.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1980931521' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:26.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:25 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:26.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1980931521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi072-82404-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:26.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:25 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi072-82404-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:26.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:26.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:25 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:26.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-47"}]: dispatch 2024-03-26T02:08:26.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:25 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-47"}]: dispatch 2024-03-26T02:08:26.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:25 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-47"}]': finished 2024-03-26T02:08:26.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:25 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi072-82404-62"}]': finished 2024-03-26T02:08:26.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:25 smithi173 ceph-mon[33839]: osdmap e339: 8 total, 8 up, 8 in 2024-03-26T02:08:26.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1980931521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:26.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:25 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:26.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1980931521' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:26.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:25 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:26.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1980931521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi072-82404-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:26.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:25 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi072-82404-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:26.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:26.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:25 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:26.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-47"}]: dispatch 2024-03-26T02:08:26.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:25 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-47"}]: dispatch 2024-03-26T02:08:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:26 smithi072 ceph-mon[30177]: pgmap v469: 324 pgs: 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T02:08:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:26 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi072-82404-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:08:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:26 smithi072 ceph-mon[30177]: osdmap e340: 8 total, 8 up, 8 in 2024-03-26T02:08:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:26 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1980931521' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi072-82404-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:26 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi072-82404-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:26 smithi072 ceph-mon[41619]: pgmap v469: 324 pgs: 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T02:08:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:26 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi072-82404-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:08:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:26 smithi072 ceph-mon[41619]: osdmap e340: 8 total, 8 up, 8 in 2024-03-26T02:08:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:26 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1980931521' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi072-82404-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:26 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi072-82404-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:27.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:26 smithi173 ceph-mon[33839]: pgmap v469: 324 pgs: 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T02:08:27.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:26 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi072-82404-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:08:27.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:26 smithi173 ceph-mon[33839]: osdmap e340: 8 total, 8 up, 8 in 2024-03-26T02:08:27.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:26 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1980931521' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi072-82404-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:27.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:26 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi072-82404-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:28.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:27 smithi072 ceph-mon[30177]: osdmap e341: 8 total, 8 up, 8 in 2024-03-26T02:08:28.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:27 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:08:28.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:27 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:08:28.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:27 smithi072 ceph-mon[41619]: osdmap e341: 8 total, 8 up, 8 in 2024-03-26T02:08:28.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:27 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:08:28.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:27 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:08:28.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:27 smithi173 ceph-mon[33839]: osdmap e341: 8 total, 8 up, 8 in 2024-03-26T02:08:28.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:27 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:08:28.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:27 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:08:29.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:28 smithi072 ceph-mon[30177]: pgmap v472: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T02:08:29.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:28 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi072-82404-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi072-82404-63"}]': finished 2024-03-26T02:08:29.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:28 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:08:29.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:28 smithi072 ceph-mon[30177]: osdmap e342: 8 total, 8 up, 8 in 2024-03-26T02:08:29.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:28 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:08:29.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:28 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:08:29.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:28 smithi072 ceph-mon[41619]: pgmap v472: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T02:08:29.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:28 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi072-82404-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi072-82404-63"}]': finished 2024-03-26T02:08:29.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:28 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:08:29.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:28 smithi072 ceph-mon[41619]: osdmap e342: 8 total, 8 up, 8 in 2024-03-26T02:08:29.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:28 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:08:29.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:28 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:08:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:28 smithi173 ceph-mon[33839]: pgmap v472: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T02:08:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:28 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi072-82404-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi072-82404-63"}]': finished 2024-03-26T02:08:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:28 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:08:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:28 smithi173 ceph-mon[33839]: osdmap e342: 8 total, 8 up, 8 in 2024-03-26T02:08:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:28 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:08:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:28 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:08:30.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:29 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:08:30.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:29 smithi072 ceph-mon[30177]: osdmap e343: 8 total, 8 up, 8 in 2024-03-26T02:08:30.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:29 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-49"}]: dispatch 2024-03-26T02:08:30.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:29 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-49"}]: dispatch 2024-03-26T02:08:30.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:29 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-49"}]': finished 2024-03-26T02:08:30.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:29 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:08:30.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:29 smithi072 ceph-mon[41619]: osdmap e343: 8 total, 8 up, 8 in 2024-03-26T02:08:30.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:29 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-49"}]: dispatch 2024-03-26T02:08:30.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:29 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-49"}]: dispatch 2024-03-26T02:08:30.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:29 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-49"}]': finished 2024-03-26T02:08:30.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:29 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:08:30.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:29 smithi173 ceph-mon[33839]: osdmap e343: 8 total, 8 up, 8 in 2024-03-26T02:08:30.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:29 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-49"}]: dispatch 2024-03-26T02:08:30.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:29 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-49"}]: dispatch 2024-03-26T02:08:30.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:29 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-49"}]': finished 2024-03-26T02:08:31.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:08:30 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:08:31.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:30 smithi173 ceph-mon[33839]: pgmap v475: 332 pgs: 40 unknown, 292 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:08:31.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-49", "mode": "readproxy"}]: dispatch 2024-03-26T02:08:31.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:30 smithi173 ceph-mon[33839]: osdmap e344: 8 total, 8 up, 8 in 2024-03-26T02:08:31.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1980931521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-49", "mode": "readproxy"}]: dispatch 2024-03-26T02:08:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:30 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:30 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:08:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-49", "mode": "readproxy"}]': finished 2024-03-26T02:08:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi072-82404-63"}]': finished 2024-03-26T02:08:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:30 smithi173 ceph-mon[33839]: osdmap e345: 8 total, 8 up, 8 in 2024-03-26T02:08:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1980931521' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:31.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:30 smithi072 ceph-mon[41619]: pgmap v475: 332 pgs: 40 unknown, 292 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:08:31.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-49", "mode": "readproxy"}]: dispatch 2024-03-26T02:08:31.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:30 smithi072 ceph-mon[41619]: osdmap e344: 8 total, 8 up, 8 in 2024-03-26T02:08:31.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1980931521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:31.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-49", "mode": "readproxy"}]: dispatch 2024-03-26T02:08:31.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:31.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:30 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:31.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:30 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:08:31.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-49", "mode": "readproxy"}]': finished 2024-03-26T02:08:31.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi072-82404-63"}]': finished 2024-03-26T02:08:31.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:30 smithi072 ceph-mon[41619]: osdmap e345: 8 total, 8 up, 8 in 2024-03-26T02:08:31.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1980931521' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:31.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:31.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:30 smithi072 ceph-mon[30177]: pgmap v475: 332 pgs: 40 unknown, 292 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:08:31.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-49", "mode": "readproxy"}]: dispatch 2024-03-26T02:08:31.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:30 smithi072 ceph-mon[30177]: osdmap e344: 8 total, 8 up, 8 in 2024-03-26T02:08:31.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1980931521' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:31.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-49", "mode": "readproxy"}]: dispatch 2024-03-26T02:08:31.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:31.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:30 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:31.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:30 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:08:31.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-49", "mode": "readproxy"}]': finished 2024-03-26T02:08:31.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi072-82404-63"}]': finished 2024-03-26T02:08:31.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:30 smithi072 ceph-mon[30177]: osdmap e345: 8 total, 8 up, 8 in 2024-03-26T02:08:31.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1980931521' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:31.221 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi072-82404-63"}]: dispatch 2024-03-26T02:08:32.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:31 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:08:32.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:31 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi072-82404-63"}]': finished 2024-03-26T02:08:32.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:31 smithi072 ceph-mon[41619]: osdmap e346: 8 total, 8 up, 8 in 2024-03-26T02:08:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:31 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:08:32.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:31 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi072-82404-63"}]': finished 2024-03-26T02:08:32.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:31 smithi072 ceph-mon[30177]: osdmap e346: 8 total, 8 up, 8 in 2024-03-26T02:08:32.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:31 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:08:32.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:31 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi072-82404-63"}]': finished 2024-03-26T02:08:32.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:31 smithi173 ceph-mon[33839]: osdmap e346: 8 total, 8 up, 8 in 2024-03-26T02:08:33.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:32 smithi072 ceph-mon[41619]: pgmap v478: 324 pgs: 324 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:08:33.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi072-82404-64"}]: dispatch 2024-03-26T02:08:33.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi072-82404-64"}]: dispatch 2024-03-26T02:08:33.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi072-82404-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:33.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi072-82404-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:08:33.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:32 smithi072 ceph-mon[30177]: pgmap v478: 324 pgs: 324 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:08:33.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi072-82404-64"}]: dispatch 2024-03-26T02:08:33.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi072-82404-64"}]: dispatch 2024-03-26T02:08:33.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi072-82404-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:33.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi072-82404-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:08:33.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:32 smithi173 ceph-mon[33839]: pgmap v478: 324 pgs: 324 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:08:33.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi072-82404-64"}]: dispatch 2024-03-26T02:08:33.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi072-82404-64"}]: dispatch 2024-03-26T02:08:33.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi072-82404-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:33.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi072-82404-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:08:34.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:34 smithi072 ceph-mon[30177]: osdmap e347: 8 total, 8 up, 8 in 2024-03-26T02:08:34.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:34 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi072-82404-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi072-82404-64"}]: dispatch 2024-03-26T02:08:34.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:34 smithi072 ceph-mon[41619]: osdmap e347: 8 total, 8 up, 8 in 2024-03-26T02:08:34.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:34 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi072-82404-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi072-82404-64"}]: dispatch 2024-03-26T02:08:34.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:34 smithi173 ceph-mon[33839]: osdmap e347: 8 total, 8 up, 8 in 2024-03-26T02:08:34.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:34 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi072-82404-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi072-82404-64"}]: dispatch 2024-03-26T02:08:35.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:08:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:08:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:08:35.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:35 smithi072 ceph-mon[30177]: pgmap v481: 324 pgs: 324 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:08:35.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:35 smithi072 ceph-mon[30177]: osdmap e348: 8 total, 8 up, 8 in 2024-03-26T02:08:35.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:35 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi072-82404-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi072-82404-64"}]': finished 2024-03-26T02:08:35.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:35 smithi072 ceph-mon[30177]: osdmap e349: 8 total, 8 up, 8 in 2024-03-26T02:08:35.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:35 smithi072 ceph-mon[41619]: pgmap v481: 324 pgs: 324 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:08:35.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:35 smithi072 ceph-mon[41619]: osdmap e348: 8 total, 8 up, 8 in 2024-03-26T02:08:35.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:35 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi072-82404-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi072-82404-64"}]': finished 2024-03-26T02:08:35.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:35 smithi072 ceph-mon[41619]: osdmap e349: 8 total, 8 up, 8 in 2024-03-26T02:08:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:35 smithi173 ceph-mon[33839]: pgmap v481: 324 pgs: 324 active+clean; 8.4 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:08:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:35 smithi173 ceph-mon[33839]: osdmap e348: 8 total, 8 up, 8 in 2024-03-26T02:08:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:35 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi072-82404-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi072-82404-64"}]': finished 2024-03-26T02:08:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:35 smithi173 ceph-mon[33839]: osdmap e349: 8 total, 8 up, 8 in 2024-03-26T02:08:36.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:36 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:36.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:36 smithi072 ceph-mon[30177]: osdmap e350: 8 total, 8 up, 8 in 2024-03-26T02:08:36.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:36 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:36.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:36 smithi072 ceph-mon[41619]: osdmap e350: 8 total, 8 up, 8 in 2024-03-26T02:08:36.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:36 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:36.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:36 smithi173 ceph-mon[33839]: osdmap e350: 8 total, 8 up, 8 in 2024-03-26T02:08:37.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:37 smithi072 ceph-mon[30177]: pgmap v484: 332 pgs: 8 creating+peering, 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T02:08:37.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:37 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:08:37.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:37 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:08:37.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:37 smithi072 ceph-mon[30177]: osdmap e351: 8 total, 8 up, 8 in 2024-03-26T02:08:37.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:37 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi072-82404-64"}]: dispatch 2024-03-26T02:08:37.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:37 smithi072 ceph-mon[41619]: pgmap v484: 332 pgs: 8 creating+peering, 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T02:08:37.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:37 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:08:37.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:37 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:08:37.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:37 smithi072 ceph-mon[41619]: osdmap e351: 8 total, 8 up, 8 in 2024-03-26T02:08:37.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:37 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi072-82404-64"}]: dispatch 2024-03-26T02:08:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:37 smithi173 ceph-mon[33839]: pgmap v484: 332 pgs: 8 creating+peering, 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T02:08:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:37 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:08:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:37 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:08:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:37 smithi173 ceph-mon[33839]: osdmap e351: 8 total, 8 up, 8 in 2024-03-26T02:08:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:37 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi072-82404-64"}]: dispatch 2024-03-26T02:08:39.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:38 smithi173 ceph-mon[33839]: pgmap v487: 324 pgs: 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T02:08:39.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:38 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi072-82404-64"}]': finished 2024-03-26T02:08:39.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:38 smithi173 ceph-mon[33839]: osdmap e352: 8 total, 8 up, 8 in 2024-03-26T02:08:39.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:38 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi072-82404-64"}]: dispatch 2024-03-26T02:08:39.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:38 smithi072 ceph-mon[30177]: pgmap v487: 324 pgs: 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T02:08:39.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:38 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi072-82404-64"}]': finished 2024-03-26T02:08:39.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:38 smithi072 ceph-mon[30177]: osdmap e352: 8 total, 8 up, 8 in 2024-03-26T02:08:39.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:38 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi072-82404-64"}]: dispatch 2024-03-26T02:08:39.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:38 smithi072 ceph-mon[41619]: pgmap v487: 324 pgs: 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T02:08:39.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:38 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi072-82404-64"}]': finished 2024-03-26T02:08:39.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:38 smithi072 ceph-mon[41619]: osdmap e352: 8 total, 8 up, 8 in 2024-03-26T02:08:39.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:38 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi072-82404-64"}]: dispatch 2024-03-26T02:08:40.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi072-82404-64"}]': finished 2024-03-26T02:08:40.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:39 smithi173 ceph-mon[33839]: osdmap e353: 8 total, 8 up, 8 in 2024-03-26T02:08:40.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/832424760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:40.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:39 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:40.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/832424760' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:40.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:39 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:40.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/832424760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi072-82404-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:40.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:39 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi072-82404-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:40.370 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi072-82404-64"}]': finished 2024-03-26T02:08:40.371 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:39 smithi072 ceph-mon[30177]: osdmap e353: 8 total, 8 up, 8 in 2024-03-26T02:08:40.371 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/832424760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:40.371 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:39 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:40.371 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/832424760' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:40.371 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:39 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:40.371 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/832424760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi072-82404-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:40.371 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:39 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi072-82404-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:40.371 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2905435563' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi072-82404-64"}]': finished 2024-03-26T02:08:40.371 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:39 smithi072 ceph-mon[41619]: osdmap e353: 8 total, 8 up, 8 in 2024-03-26T02:08:40.372 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/832424760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:40.372 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:39 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:40.372 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/832424760' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:40.372 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:39 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:40.372 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/832424760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi072-82404-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:40.372 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:39 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi072-82404-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:41.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:08:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:08:41.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:41 smithi173 ceph-mon[33839]: pgmap v490: 324 pgs: 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:08:41.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:41 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi072-82404-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:08:41.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:41 smithi173 ceph-mon[33839]: osdmap e354: 8 total, 8 up, 8 in 2024-03-26T02:08:41.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:41 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/832424760' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi072-82404-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:41.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:41 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi072-82404-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:41.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:41 smithi072 ceph-mon[41619]: pgmap v490: 324 pgs: 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:08:41.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:41 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi072-82404-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:08:41.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:41 smithi072 ceph-mon[41619]: osdmap e354: 8 total, 8 up, 8 in 2024-03-26T02:08:41.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:41 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/832424760' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi072-82404-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:41.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:41 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi072-82404-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:41.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:41 smithi072 ceph-mon[30177]: pgmap v490: 324 pgs: 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:08:41.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:41 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi072-82404-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:08:41.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:41 smithi072 ceph-mon[30177]: osdmap e354: 8 total, 8 up, 8 in 2024-03-26T02:08:41.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:41 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/832424760' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi072-82404-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:41.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:41 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi072-82404-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:42 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:08:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:42 smithi173 ceph-mon[33839]: osdmap e355: 8 total, 8 up, 8 in 2024-03-26T02:08:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:42 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:42 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:42 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi072-82404-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi072-82404-65"}]': finished 2024-03-26T02:08:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:42 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:08:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:42 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-49"}]: dispatch 2024-03-26T02:08:42.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:42 smithi173 ceph-mon[33839]: osdmap e356: 8 total, 8 up, 8 in 2024-03-26T02:08:42.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:42 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-49"}]: dispatch 2024-03-26T02:08:42.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:42 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:42.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:42 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:08:42.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:42 smithi072 ceph-mon[30177]: osdmap e355: 8 total, 8 up, 8 in 2024-03-26T02:08:42.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:42 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:42.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:42 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:42.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:42 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi072-82404-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi072-82404-65"}]': finished 2024-03-26T02:08:42.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:42 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:08:42.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:42 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-49"}]: dispatch 2024-03-26T02:08:42.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:42 smithi072 ceph-mon[30177]: osdmap e356: 8 total, 8 up, 8 in 2024-03-26T02:08:42.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:42 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-49"}]: dispatch 2024-03-26T02:08:42.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:42 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:42.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:42 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:08:42.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:42 smithi072 ceph-mon[41619]: osdmap e355: 8 total, 8 up, 8 in 2024-03-26T02:08:42.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:42 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:42.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:42 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:42.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:42 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi072-82404-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi072-82404-65"}]': finished 2024-03-26T02:08:42.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:42 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:08:42.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:42 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-49"}]: dispatch 2024-03-26T02:08:42.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:42 smithi072 ceph-mon[41619]: osdmap e356: 8 total, 8 up, 8 in 2024-03-26T02:08:42.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:42 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-49"}]: dispatch 2024-03-26T02:08:42.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:42 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:08:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:43 smithi173 ceph-mon[33839]: pgmap v493: 324 pgs: 324 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:08:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:43 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:43 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:43.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:43 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:08:43.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:43 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:08:43.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:43 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-49"}]': finished 2024-03-26T02:08:43.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:43 smithi173 ceph-mon[33839]: osdmap e357: 8 total, 8 up, 8 in 2024-03-26T02:08:43.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:43 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:43.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:43 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:43.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:43 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-49"}]: dispatch 2024-03-26T02:08:43.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:43 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-49"}]: dispatch 2024-03-26T02:08:43.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:43 smithi072 ceph-mon[30177]: pgmap v493: 324 pgs: 324 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:08:43.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:43 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:43.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:43 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:43.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:43 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:08:43.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:43 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:08:43.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:43 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-49"}]': finished 2024-03-26T02:08:43.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:43 smithi072 ceph-mon[30177]: osdmap e357: 8 total, 8 up, 8 in 2024-03-26T02:08:43.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:43 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:43.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:43 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:43.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:43 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-49"}]: dispatch 2024-03-26T02:08:43.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:43 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-49"}]: dispatch 2024-03-26T02:08:43.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:43 smithi072 ceph-mon[41619]: pgmap v493: 324 pgs: 324 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:08:43.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:43 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:08:43.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:43 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:08:43.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:43 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:08:43.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:43 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:08:43.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:43 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-49"}]': finished 2024-03-26T02:08:43.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:43 smithi072 ceph-mon[41619]: osdmap e357: 8 total, 8 up, 8 in 2024-03-26T02:08:43.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:43 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:43.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:43 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:08:43.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:43 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-49"}]: dispatch 2024-03-26T02:08:43.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:43 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-49"}]: dispatch 2024-03-26T02:08:43.991 INFO:tasks.workunit.client.0.smithi072.stdout:163:head 2024-03-26T02:08:43.992 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:5d165639:::164:head 2024-03-26T02:08:43.992 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:f43765fc:::165:head 2024-03-26T02:08:43.992 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:b4c720e9:::166:head 2024-03-26T02:08:43.992 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:e694b040:::167:head 2024-03-26T02:08:43.992 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:afa38db2:::168:head 2024-03-26T02:08:43.992 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:77ba9f53:::169:head 2024-03-26T02:08:43.992 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:87495034:::170:head 2024-03-26T02:08:43.992 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:7c96bf0e:::171:head 2024-03-26T02:08:43.992 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:dbe346cc:::172:head 2024-03-26T02:08:43.992 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:e943ec24:::173:head 2024-03-26T02:08:43.992 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:f97a9c0c:::174:head 2024-03-26T02:08:43.992 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:6f26e74d:::175:head 2024-03-26T02:08:43.993 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:4f95e106:::176:head 2024-03-26T02:08:43.993 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:0e6f2f8f:::177:head 2024-03-26T02:08:43.993 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:05db05f1:::178:head 2024-03-26T02:08:43.993 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:38a78d66:::179:head 2024-03-26T02:08:43.993 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:d095610b:::180:head 2024-03-26T02:08:43.993 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:a1a9d709:::181:head 2024-03-26T02:08:43.993 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:1e5d39db:::182:head 2024-03-26T02:08:43.993 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:f7df4fb9:::183:head 2024-03-26T02:08:43.993 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:03a7f161:::184:head 2024-03-26T02:08:43.993 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:ba70721e:::185:head 2024-03-26T02:08:43.993 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:28e5662d:::186:head 2024-03-26T02:08:43.993 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:973d52de:::187:head 2024-03-26T02:08:43.993 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:4303eb1c:::188:head 2024-03-26T02:08:43.993 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:b990b48e:::189:head 2024-03-26T02:08:43.994 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:29b8165b:::190:head 2024-03-26T02:08:43.994 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:3547f197:::191:head 2024-03-26T02:08:43.994 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:7e260936:::192:head 2024-03-26T02:08:43.994 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:1abec7b1:::193:head 2024-03-26T02:08:43.994 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:10fdda93:::194:head 2024-03-26T02:08:43.994 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:15817eea:::195:head 2024-03-26T02:08:43.994 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:770bab57:::196:head 2024-03-26T02:08:43.994 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:ed9e13e7:::197:head 2024-03-26T02:08:43.994 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:71471a8f:::198:head 2024-03-26T02:08:43.994 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: checking for 266:10fb1d02:::199:head 2024-03-26T02:08:43.995 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8052 ms) 2024-03-26T02:08:43.995 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-03-26T02:08:43.995 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: got ls 1711418878,0 2024-03-26T02:08:43.995 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: first is 1711418878 2024-03-26T02:08:43.995 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: got ls 1711418878,0 2024-03-26T02:08:43.995 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: got ls 1711418878,0 2024-03-26T02:08:43.995 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: got ls 1711418878,1711418880,1711418881,0 2024-03-26T02:08:43.995 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: got ls 1711418878,1711418880,1711418881,0 2024-03-26T02:08:43.995 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: got ls 1711418878,1711418880,1711418881,0 2024-03-26T02:08:43.995 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: got ls 1711418878,1711418880,1711418881,1711418883,1711418884,0 2024-03-26T02:08:43.995 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: got ls 1711418878,1711418880,1711418881,1711418883,1711418884,0 2024-03-26T02:08:43.995 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: got ls 1711418878,1711418880,1711418881,1711418883,1711418884,0 2024-03-26T02:08:43.995 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: got ls 1711418878,1711418880,1711418881,1711418883,1711418884,1711418886,1711418887,0 2024-03-26T02:08:43.996 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: got ls 1711418878,1711418880,1711418881,1711418883,1711418884,1711418886,1711418887,0 2024-03-26T02:08:43.996 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: got ls 1711418878,1711418880,1711418881,1711418883,1711418884,1711418886,1711418887,0 2024-03-26T02:08:43.996 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: got ls 1711418881,1711418883,1711418884,1711418886,1711418887,1711418889,1711418890,0 2024-03-26T02:08:43.996 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: first now 1711418881, trimmed 2024-03-26T02:08:43.996 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (21081 ms) 2024-03-26T02:08:43.996 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-03-26T02:08:43.996 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: foo0 2024-03-26T02:08:43.996 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-26T02:08:43.996 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14042 ms) 2024-03-26T02:08:43.996 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-03-26T02:08:45.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:44 smithi173 ceph-mon[33839]: pgmap v496: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:08:45.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:44 smithi173 ceph-mon[33839]: osdmap e358: 8 total, 8 up, 8 in 2024-03-26T02:08:45.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:44 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/832424760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:45.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:44 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:45.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:44 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:45.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:08:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:08:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:08:45.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:44 smithi072 ceph-mon[30177]: pgmap v496: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:08:45.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:44 smithi072 ceph-mon[30177]: osdmap e358: 8 total, 8 up, 8 in 2024-03-26T02:08:45.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:44 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/832424760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:45.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:44 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:45.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:44 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:45.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:44 smithi072 ceph-mon[41619]: pgmap v496: 332 pgs: 8 unknown, 324 active+clean; 8.4 MiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:08:45.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:44 smithi072 ceph-mon[41619]: osdmap e358: 8 total, 8 up, 8 in 2024-03-26T02:08:45.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:44 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/832424760' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:45.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:44 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:45.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:44 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:46.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:46 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi072-82404-65"}]': finished 2024-03-26T02:08:46.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:46 smithi173 ceph-mon[33839]: osdmap e359: 8 total, 8 up, 8 in 2024-03-26T02:08:46.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:46 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/832424760' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:46.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:46 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:08:46.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:46 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:46.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:46 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:08:46.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:46 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi072-82404-65"}]': finished 2024-03-26T02:08:46.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:46 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:08:46.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:46 smithi173 ceph-mon[33839]: osdmap e360: 8 total, 8 up, 8 in 2024-03-26T02:08:46.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:46 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi072-82404-65"}]': finished 2024-03-26T02:08:46.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:46 smithi072 ceph-mon[30177]: osdmap e359: 8 total, 8 up, 8 in 2024-03-26T02:08:46.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:46 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/832424760' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:46.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:46 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:08:46.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:46 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:46.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:46 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:08:46.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:46 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi072-82404-65"}]': finished 2024-03-26T02:08:46.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:46 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:08:46.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:46 smithi072 ceph-mon[30177]: osdmap e360: 8 total, 8 up, 8 in 2024-03-26T02:08:46.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:46 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi072-82404-65"}]': finished 2024-03-26T02:08:46.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:46 smithi072 ceph-mon[41619]: osdmap e359: 8 total, 8 up, 8 in 2024-03-26T02:08:46.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:46 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/832424760' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:46.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:46 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:08:46.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:46 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi072-82404-65"}]: dispatch 2024-03-26T02:08:46.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:46 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:08:46.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:46 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi072-82404-65"}]': finished 2024-03-26T02:08:46.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:46 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:08:46.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:46 smithi072 ceph-mon[41619]: osdmap e360: 8 total, 8 up, 8 in 2024-03-26T02:08:47.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:47 smithi173 ceph-mon[33839]: pgmap v499: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:08:47.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:47 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:08:47.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:47 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:08:47.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:47 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2220841605' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:47.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:47 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:47.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:47 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2220841605' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:47.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:47 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:47.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:47 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2220841605' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi072-82404-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:47.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:47 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi072-82404-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:47.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:47 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:08:47.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:47 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi072-82404-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:08:47.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:47 smithi173 ceph-mon[33839]: osdmap e361: 8 total, 8 up, 8 in 2024-03-26T02:08:47.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:47 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-51"}]: dispatch 2024-03-26T02:08:47.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:47 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2220841605' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi072-82404-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:47.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:47 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-51"}]: dispatch 2024-03-26T02:08:47.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:47 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi072-82404-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:47.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[30177]: pgmap v499: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:08:47.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:08:47.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:08:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2220841605' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2220841605' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2220841605' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi072-82404-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi072-82404-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:08:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi072-82404-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:08:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[30177]: osdmap e361: 8 total, 8 up, 8 in 2024-03-26T02:08:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-51"}]: dispatch 2024-03-26T02:08:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2220841605' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi072-82404-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-51"}]: dispatch 2024-03-26T02:08:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi072-82404-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:47.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[41619]: pgmap v499: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:08:47.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:08:47.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:08:47.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2220841605' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:47.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:47.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2220841605' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:47.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:47.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2220841605' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi072-82404-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:47.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi072-82404-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:08:47.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:08:47.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi072-82404-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:08:47.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[41619]: osdmap e361: 8 total, 8 up, 8 in 2024-03-26T02:08:47.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-51"}]: dispatch 2024-03-26T02:08:47.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2220841605' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi072-82404-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:47.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-51"}]: dispatch 2024-03-26T02:08:47.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:47 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi072-82404-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:49 smithi173 ceph-mon[33839]: pgmap v502: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:08:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:49 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-51"}]': finished 2024-03-26T02:08:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:49 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-51", "mode": "writeback"}]: dispatch 2024-03-26T02:08:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:49 smithi173 ceph-mon[33839]: osdmap e362: 8 total, 8 up, 8 in 2024-03-26T02:08:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:49 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-51", "mode": "writeback"}]: dispatch 2024-03-26T02:08:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:49 smithi072 ceph-mon[30177]: pgmap v502: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:08:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:49 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-51"}]': finished 2024-03-26T02:08:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:49 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-51", "mode": "writeback"}]: dispatch 2024-03-26T02:08:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:49 smithi072 ceph-mon[30177]: osdmap e362: 8 total, 8 up, 8 in 2024-03-26T02:08:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:49 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-51", "mode": "writeback"}]: dispatch 2024-03-26T02:08:49.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:49 smithi072 ceph-mon[41619]: pgmap v502: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:08:49.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:49 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-51"}]': finished 2024-03-26T02:08:49.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:49 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-51", "mode": "writeback"}]: dispatch 2024-03-26T02:08:49.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:49 smithi072 ceph-mon[41619]: osdmap e362: 8 total, 8 up, 8 in 2024-03-26T02:08:49.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:49 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-51", "mode": "writeback"}]: dispatch 2024-03-26T02:08:50.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:50 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:08:50.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:50 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi072-82404-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi072-82404-66"}]': finished 2024-03-26T02:08:50.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:50 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-51", "mode": "writeback"}]': finished 2024-03-26T02:08:50.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:50 smithi173 ceph-mon[33839]: osdmap e363: 8 total, 8 up, 8 in 2024-03-26T02:08:50.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:50 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:08:50.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:50 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:08:50.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:50 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:08:50.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:50 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi072-82404-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi072-82404-66"}]': finished 2024-03-26T02:08:50.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:50 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-51", "mode": "writeback"}]': finished 2024-03-26T02:08:50.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:50 smithi072 ceph-mon[41619]: osdmap e363: 8 total, 8 up, 8 in 2024-03-26T02:08:50.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:50 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:08:50.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:50 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:08:50.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:50 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:08:50.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:50 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi072-82404-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi072-82404-66"}]': finished 2024-03-26T02:08:50.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:50 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-51", "mode": "writeback"}]': finished 2024-03-26T02:08:50.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:50 smithi072 ceph-mon[30177]: osdmap e363: 8 total, 8 up, 8 in 2024-03-26T02:08:50.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:50 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:08:50.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:50 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:08:51.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:08:50 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:08:51.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:51 smithi173 ceph-mon[33839]: pgmap v505: 332 pgs: 8 unknown, 32 creating+peering, 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:08:51.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:51 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:51.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:51 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_count","val": "2"}]': finished 2024-03-26T02:08:51.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:51 smithi173 ceph-mon[33839]: osdmap e364: 8 total, 8 up, 8 in 2024-03-26T02:08:51.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:51 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:08:51.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:51 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:08:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:51 smithi072 ceph-mon[30177]: pgmap v505: 332 pgs: 8 unknown, 32 creating+peering, 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:08:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:51 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:51 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_count","val": "2"}]': finished 2024-03-26T02:08:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:51 smithi072 ceph-mon[30177]: osdmap e364: 8 total, 8 up, 8 in 2024-03-26T02:08:51.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:51 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:08:51.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:51 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:08:51.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:51 smithi072 ceph-mon[41619]: pgmap v505: 332 pgs: 8 unknown, 32 creating+peering, 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:08:51.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:51 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:51.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:51 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_count","val": "2"}]': finished 2024-03-26T02:08:51.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:51 smithi072 ceph-mon[41619]: osdmap e364: 8 total, 8 up, 8 in 2024-03-26T02:08:51.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:51 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:08:51.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:51 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:08:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:52 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:08:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:52 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:08:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:52 smithi173 ceph-mon[33839]: osdmap e365: 8 total, 8 up, 8 in 2024-03-26T02:08:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:52 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:08:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:52 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2220841605' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:52 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:08:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:52 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:52.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:08:52.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:52 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:08:52.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:52 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:08:52.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:52 smithi072 ceph-mon[30177]: osdmap e365: 8 total, 8 up, 8 in 2024-03-26T02:08:52.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:52 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:08:52.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:52 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2220841605' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:52.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:52 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:08:52.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:52 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:52.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:08:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:52 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:08:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:52 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:08:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:52 smithi072 ceph-mon[41619]: osdmap e365: 8 total, 8 up, 8 in 2024-03-26T02:08:52.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:52 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:08:52.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:52 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2220841605' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:52.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:52 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:08:52.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:52 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:52.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:08:53.041 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3020 ms) 2024-03-26T02:08:53.041 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-03-26T02:08:53.041 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7031 ms) 2024-03-26T02:08:53.042 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-03-26T02:08:53.042 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7018 ms) 2024-03-26T02:08:53.042 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-03-26T02:08:53.042 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7032 ms) 2024-03-26T02:08:53.042 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-03-26T02:08:53.042 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7036 ms) 2024-03-26T02:08:53.042 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-03-26T02:08:53.042 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7025 ms) 2024-03-26T02:08:53.042 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-03-26T02:08:53.042 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7040 ms) 2024-03-26T02:08:53.042 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (139323 ms total) 2024-03-26T02:08:53.042 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: 2024-03-26T02:08:53.042 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-03-26T02:08:53.043 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (303993 ms total) 2024-03-26T02:08:53.043 INFO:tasks.workunit.client.0.smithi072.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-03-26T02:08:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:53 smithi173 ceph-mon[33839]: pgmap v508: 324 pgs: 324 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T02:08:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:53 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:08:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:53 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T02:08:53.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:53 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi072-82404-66"}]': finished 2024-03-26T02:08:53.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:53 smithi173 ceph-mon[33839]: osdmap e366: 8 total, 8 up, 8 in 2024-03-26T02:08:53.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:53 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T02:08:53.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:53 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2220841605' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:53.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:53 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T02:08:53.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:53 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:53 smithi072 ceph-mon[30177]: pgmap v508: 324 pgs: 324 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T02:08:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:53 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:08:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:53 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T02:08:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:53 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi072-82404-66"}]': finished 2024-03-26T02:08:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:53 smithi072 ceph-mon[30177]: osdmap e366: 8 total, 8 up, 8 in 2024-03-26T02:08:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:53 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T02:08:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:53 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2220841605' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:53 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T02:08:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:53 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:53 smithi072 ceph-mon[41619]: pgmap v508: 324 pgs: 324 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T02:08:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:53 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:08:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:53 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T02:08:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:53 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi072-82404-66"}]': finished 2024-03-26T02:08:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:53 smithi072 ceph-mon[41619]: osdmap e366: 8 total, 8 up, 8 in 2024-03-26T02:08:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:53 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T02:08:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:53 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2220841605' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:53.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:53 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T02:08:53.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:53 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi072-82404-66"}]: dispatch 2024-03-26T02:08:54.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:54 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T02:08:54.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:54 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi072-82404-66"}]': finished 2024-03-26T02:08:54.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:54 smithi173 ceph-mon[33839]: osdmap e367: 8 total, 8 up, 8 in 2024-03-26T02:08:54.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:54 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T02:08:54.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:54 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T02:08:54.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:54 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "target_max_objects","val": "1"}]': finished 2024-03-26T02:08:54.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:54 smithi173 ceph-mon[33839]: osdmap e368: 8 total, 8 up, 8 in 2024-03-26T02:08:54.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:54 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T02:08:54.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:54 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi072-82404-66"}]': finished 2024-03-26T02:08:54.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:54 smithi072 ceph-mon[30177]: osdmap e367: 8 total, 8 up, 8 in 2024-03-26T02:08:54.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:54 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T02:08:54.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:54 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T02:08:54.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:54 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "target_max_objects","val": "1"}]': finished 2024-03-26T02:08:54.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:54 smithi072 ceph-mon[30177]: osdmap e368: 8 total, 8 up, 8 in 2024-03-26T02:08:54.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:54 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T02:08:54.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:54 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi072-82404-66"}]': finished 2024-03-26T02:08:54.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:54 smithi072 ceph-mon[41619]: osdmap e367: 8 total, 8 up, 8 in 2024-03-26T02:08:54.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:54 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T02:08:54.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:54 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T02:08:54.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:54 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-51","var": "target_max_objects","val": "1"}]': finished 2024-03-26T02:08:54.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:54 smithi072 ceph-mon[41619]: osdmap e368: 8 total, 8 up, 8 in 2024-03-26T02:08:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:55 smithi173 ceph-mon[33839]: pgmap v511: 324 pgs: 324 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T02:08:55.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:55 smithi072 ceph-mon[30177]: pgmap v511: 324 pgs: 324 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T02:08:55.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:55 smithi072 ceph-mon[41619]: pgmap v511: 324 pgs: 324 active+clean; 8.4 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T02:08:55.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:08:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:08:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:08:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:56 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:56.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:56 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:56.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:56 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:08:57.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:57 smithi173 ceph-mon[33839]: pgmap v513: 324 pgs: 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:08:57.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:57 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T02:08:57.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:57 smithi072 ceph-mon[30177]: pgmap v513: 324 pgs: 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:08:57.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:57 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T02:08:57.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:57 smithi072 ceph-mon[41619]: pgmap v513: 324 pgs: 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:08:57.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:57 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T02:08:59.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:08:59 smithi173 ceph-mon[33839]: pgmap v514: 324 pgs: 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:08:59.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:08:59 smithi072 ceph-mon[41619]: pgmap v514: 324 pgs: 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:08:59.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:08:59 smithi072 ceph-mon[30177]: pgmap v514: 324 pgs: 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:09:01.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:01 smithi173 ceph-mon[33839]: pgmap v515: 324 pgs: 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 678 B/s rd, 0 B/s wr, 0 op/s 2024-03-26T02:09:01.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:09:00 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:09:01.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:01 smithi072 ceph-mon[30177]: pgmap v515: 324 pgs: 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 678 B/s rd, 0 B/s wr, 0 op/s 2024-03-26T02:09:01.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:01 smithi072 ceph-mon[41619]: pgmap v515: 324 pgs: 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 678 B/s rd, 0 B/s wr, 0 op/s 2024-03-26T02:09:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:02 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:09:02.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:02 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:09:02.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:02 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:09:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:03 smithi173 ceph-mon[33839]: pgmap v516: 324 pgs: 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:09:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:03 smithi072 ceph-mon[30177]: pgmap v516: 324 pgs: 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:09:03.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:03 smithi072 ceph-mon[41619]: pgmap v516: 324 pgs: 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:09:04.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:04 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:04.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:04 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:04.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:04 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:04.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:04 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:04.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:04 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:04.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:04 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:05.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:09:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:09:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:09:05.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:05 smithi072 ceph-mon[30177]: pgmap v517: 324 pgs: 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:09:05.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:09:05.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:05 smithi072 ceph-mon[30177]: osdmap e369: 8 total, 8 up, 8 in 2024-03-26T02:09:05.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-51"}]: dispatch 2024-03-26T02:09:05.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-51"}]: dispatch 2024-03-26T02:09:05.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:05 smithi072 ceph-mon[41619]: pgmap v517: 324 pgs: 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:09:05.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:09:05.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:05 smithi072 ceph-mon[41619]: osdmap e369: 8 total, 8 up, 8 in 2024-03-26T02:09:05.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-51"}]: dispatch 2024-03-26T02:09:05.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-51"}]: dispatch 2024-03-26T02:09:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:05 smithi173 ceph-mon[33839]: pgmap v517: 324 pgs: 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:09:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:09:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:05 smithi173 ceph-mon[33839]: osdmap e369: 8 total, 8 up, 8 in 2024-03-26T02:09:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-51"}]: dispatch 2024-03-26T02:09:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-51"}]: dispatch 2024-03-26T02:09:06.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:06 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-51"}]': finished 2024-03-26T02:09:06.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:06 smithi072 ceph-mon[41619]: osdmap e370: 8 total, 8 up, 8 in 2024-03-26T02:09:06.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:06.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:06 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:06.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-51"}]: dispatch 2024-03-26T02:09:06.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:06 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-51"}]: dispatch 2024-03-26T02:09:06.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:06 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-51"}]': finished 2024-03-26T02:09:06.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:06 smithi072 ceph-mon[30177]: osdmap e370: 8 total, 8 up, 8 in 2024-03-26T02:09:06.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:06.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:06 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:06.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-51"}]: dispatch 2024-03-26T02:09:06.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:06 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-51"}]: dispatch 2024-03-26T02:09:06.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:06 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-51"}]': finished 2024-03-26T02:09:06.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:06 smithi173 ceph-mon[33839]: osdmap e370: 8 total, 8 up, 8 in 2024-03-26T02:09:06.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:06.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:06 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:06.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-51"}]: dispatch 2024-03-26T02:09:06.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:06 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-51"}]: dispatch 2024-03-26T02:09:07.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:07 smithi072 ceph-mon[30177]: pgmap v520: 324 pgs: 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:09:07.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:07 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T02:09:07.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:07 smithi072 ceph-mon[30177]: osdmap e371: 8 total, 8 up, 8 in 2024-03-26T02:09:07.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:07 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:09:07.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:07 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:09:07.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:07 smithi072 ceph-mon[41619]: pgmap v520: 324 pgs: 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:09:07.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:07 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T02:09:07.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:07 smithi072 ceph-mon[41619]: osdmap e371: 8 total, 8 up, 8 in 2024-03-26T02:09:07.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:07 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:09:07.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:07 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:09:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:07 smithi173 ceph-mon[33839]: pgmap v520: 324 pgs: 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:09:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:07 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T02:09:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:07 smithi173 ceph-mon[33839]: osdmap e371: 8 total, 8 up, 8 in 2024-03-26T02:09:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:07 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:09:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:07 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:09:08.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:08 smithi072 ceph-mon[41619]: osdmap e372: 8 total, 8 up, 8 in 2024-03-26T02:09:08.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:08.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:08.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:08 smithi072 ceph-mon[30177]: osdmap e372: 8 total, 8 up, 8 in 2024-03-26T02:09:08.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:08.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:08 smithi173 ceph-mon[33839]: osdmap e372: 8 total, 8 up, 8 in 2024-03-26T02:09:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:09.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:09 smithi072 ceph-mon[41619]: pgmap v523: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:09.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:09 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:09:09.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:09:09.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:09 smithi072 ceph-mon[41619]: osdmap e373: 8 total, 8 up, 8 in 2024-03-26T02:09:09.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:09.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:09.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-53"}]: dispatch 2024-03-26T02:09:09.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-53"}]: dispatch 2024-03-26T02:09:09.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:09 smithi072 ceph-mon[30177]: pgmap v523: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:09.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:09 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:09:09.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:09:09.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:09 smithi072 ceph-mon[30177]: osdmap e373: 8 total, 8 up, 8 in 2024-03-26T02:09:09.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:09.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:09.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-53"}]: dispatch 2024-03-26T02:09:09.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-53"}]: dispatch 2024-03-26T02:09:09.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:09 smithi173 ceph-mon[33839]: pgmap v523: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:09:09.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:09 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:09:09.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:09:09.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:09 smithi173 ceph-mon[33839]: osdmap e373: 8 total, 8 up, 8 in 2024-03-26T02:09:09.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:09.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:09.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-53"}]: dispatch 2024-03-26T02:09:09.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-53"}]: dispatch 2024-03-26T02:09:10.370 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:10 smithi072 ceph-mon[30177]: osdmap e374: 8 total, 8 up, 8 in 2024-03-26T02:09:10.371 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:10 smithi072 ceph-mon[41619]: osdmap e374: 8 total, 8 up, 8 in 2024-03-26T02:09:10.464 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:10 smithi173 ceph-mon[33839]: osdmap e374: 8 total, 8 up, 8 in 2024-03-26T02:09:11.113 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:09:10 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:09:11.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:11 smithi072 ceph-mon[41619]: pgmap v526: 292 pgs: 292 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:09:11.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:11 smithi072 ceph-mon[41619]: osdmap e375: 8 total, 8 up, 8 in 2024-03-26T02:09:11.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:11 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:11.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:11 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:11.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:11 smithi072 ceph-mon[30177]: pgmap v526: 292 pgs: 292 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:09:11.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:11 smithi072 ceph-mon[30177]: osdmap e375: 8 total, 8 up, 8 in 2024-03-26T02:09:11.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:11 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:11.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:11 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:11.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:11 smithi173 ceph-mon[33839]: pgmap v526: 292 pgs: 292 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:09:11.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:11 smithi173 ceph-mon[33839]: osdmap e375: 8 total, 8 up, 8 in 2024-03-26T02:09:11.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:11 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:11.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:11 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:12.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:12 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:09:12.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:09:12.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:12 smithi072 ceph-mon[30177]: osdmap e376: 8 total, 8 up, 8 in 2024-03-26T02:09:12.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:09:12.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:12.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:12.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-55"}]: dispatch 2024-03-26T02:09:12.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-55"}]: dispatch 2024-03-26T02:09:12.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:12 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:09:12.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:09:12.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:12 smithi072 ceph-mon[41619]: osdmap e376: 8 total, 8 up, 8 in 2024-03-26T02:09:12.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:09:12.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:12.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:12.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-55"}]: dispatch 2024-03-26T02:09:12.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-55"}]: dispatch 2024-03-26T02:09:12.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:12 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:09:12.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:09:12.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:12 smithi173 ceph-mon[33839]: osdmap e376: 8 total, 8 up, 8 in 2024-03-26T02:09:12.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:09:12.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:12.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:12.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-55"}]: dispatch 2024-03-26T02:09:12.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-55"}]: dispatch 2024-03-26T02:09:13.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:13 smithi072 ceph-mon[30177]: pgmap v529: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T02:09:13.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:13 smithi072 ceph-mon[30177]: osdmap e377: 8 total, 8 up, 8 in 2024-03-26T02:09:13.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:13 smithi072 ceph-mon[41619]: pgmap v529: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T02:09:13.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:13 smithi072 ceph-mon[41619]: osdmap e377: 8 total, 8 up, 8 in 2024-03-26T02:09:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:13 smithi173 ceph-mon[33839]: pgmap v529: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T02:09:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:13 smithi173 ceph-mon[33839]: osdmap e377: 8 total, 8 up, 8 in 2024-03-26T02:09:14.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:14 smithi072 ceph-mon[41619]: osdmap e378: 8 total, 8 up, 8 in 2024-03-26T02:09:14.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:14 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:14.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:14 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:14.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:14 smithi072 ceph-mon[30177]: osdmap e378: 8 total, 8 up, 8 in 2024-03-26T02:09:14.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:14 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:14.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:14 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:14.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:14 smithi173 ceph-mon[33839]: osdmap e378: 8 total, 8 up, 8 in 2024-03-26T02:09:14.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:14 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:14.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:14 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:15.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:09:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:09:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:09:15.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:15 smithi072 ceph-mon[30177]: pgmap v532: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T02:09:15.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:15 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:09:15.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:09:15.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:15 smithi072 ceph-mon[30177]: osdmap e379: 8 total, 8 up, 8 in 2024-03-26T02:09:15.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:15.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:15.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-57"}]: dispatch 2024-03-26T02:09:15.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-57"}]: dispatch 2024-03-26T02:09:15.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:15 smithi072 ceph-mon[41619]: pgmap v532: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T02:09:15.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:15 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:09:15.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:09:15.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:15 smithi072 ceph-mon[41619]: osdmap e379: 8 total, 8 up, 8 in 2024-03-26T02:09:15.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:15.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:15.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-57"}]: dispatch 2024-03-26T02:09:15.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-57"}]: dispatch 2024-03-26T02:09:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:15 smithi173 ceph-mon[33839]: pgmap v532: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T02:09:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:15 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:09:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:09:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:15 smithi173 ceph-mon[33839]: osdmap e379: 8 total, 8 up, 8 in 2024-03-26T02:09:15.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:15.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:15.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-57"}]: dispatch 2024-03-26T02:09:15.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-57"}]: dispatch 2024-03-26T02:09:16.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:16 smithi072 ceph-mon[30177]: osdmap e380: 8 total, 8 up, 8 in 2024-03-26T02:09:16.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:16 smithi072 ceph-mon[41619]: osdmap e380: 8 total, 8 up, 8 in 2024-03-26T02:09:16.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:16 smithi173 ceph-mon[33839]: osdmap e380: 8 total, 8 up, 8 in 2024-03-26T02:09:17.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:17 smithi072 ceph-mon[30177]: pgmap v535: 292 pgs: 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T02:09:17.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:17 smithi072 ceph-mon[30177]: osdmap e381: 8 total, 8 up, 8 in 2024-03-26T02:09:17.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:17 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:17.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:17 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:17.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:17 smithi072 ceph-mon[41619]: pgmap v535: 292 pgs: 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T02:09:17.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:17 smithi072 ceph-mon[41619]: osdmap e381: 8 total, 8 up, 8 in 2024-03-26T02:09:17.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:17 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:17.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:17 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:17.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:17 smithi173 ceph-mon[33839]: pgmap v535: 292 pgs: 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T02:09:17.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:17 smithi173 ceph-mon[33839]: osdmap e381: 8 total, 8 up, 8 in 2024-03-26T02:09:17.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:17 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:17.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:17 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:18.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:09:18.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:18 smithi072 ceph-mon[30177]: osdmap e382: 8 total, 8 up, 8 in 2024-03-26T02:09:18.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:09:18.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:18.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:18.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-59"}]: dispatch 2024-03-26T02:09:18.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-59"}]: dispatch 2024-03-26T02:09:18.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:09:18.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:18 smithi072 ceph-mon[41619]: osdmap e382: 8 total, 8 up, 8 in 2024-03-26T02:09:18.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:09:18.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:18.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:18.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-59"}]: dispatch 2024-03-26T02:09:18.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-59"}]: dispatch 2024-03-26T02:09:18.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:09:18.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:18 smithi173 ceph-mon[33839]: osdmap e382: 8 total, 8 up, 8 in 2024-03-26T02:09:18.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:09:18.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:18.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:18.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-59"}]: dispatch 2024-03-26T02:09:18.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-59"}]: dispatch 2024-03-26T02:09:19.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:19 smithi072 ceph-mon[30177]: pgmap v538: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T02:09:19.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:19 smithi072 ceph-mon[30177]: osdmap e383: 8 total, 8 up, 8 in 2024-03-26T02:09:19.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:19 smithi072 ceph-mon[41619]: pgmap v538: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T02:09:19.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:19 smithi072 ceph-mon[41619]: osdmap e383: 8 total, 8 up, 8 in 2024-03-26T02:09:19.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:19 smithi173 ceph-mon[33839]: pgmap v538: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T02:09:19.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:19 smithi173 ceph-mon[33839]: osdmap e383: 8 total, 8 up, 8 in 2024-03-26T02:09:20.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:20 smithi072 ceph-mon[30177]: osdmap e384: 8 total, 8 up, 8 in 2024-03-26T02:09:20.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:20.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:20 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:20.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:20 smithi072 ceph-mon[41619]: osdmap e384: 8 total, 8 up, 8 in 2024-03-26T02:09:20.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:20.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:20 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:20.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:20 smithi173 ceph-mon[33839]: osdmap e384: 8 total, 8 up, 8 in 2024-03-26T02:09:20.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:20.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:20 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:21.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:09:20 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:09:21.182 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:21 smithi173 ceph-mon[33839]: pgmap v541: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:09:21.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:21 smithi072 ceph-mon[30177]: pgmap v541: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:09:21.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:21 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:09:21.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:21 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:09:21.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:21 smithi072 ceph-mon[30177]: osdmap e385: 8 total, 8 up, 8 in 2024-03-26T02:09:21.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:09:21.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:09:21.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:21 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:09:21.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:21 smithi072 ceph-mon[41619]: pgmap v541: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:09:21.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:21 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:09:21.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:21 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:09:21.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:21 smithi072 ceph-mon[41619]: osdmap e385: 8 total, 8 up, 8 in 2024-03-26T02:09:21.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:09:21.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:09:21.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:21 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:09:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:21 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:09:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:21 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:09:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:21 smithi173 ceph-mon[33839]: osdmap e385: 8 total, 8 up, 8 in 2024-03-26T02:09:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:09:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:09:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:21 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:09:22.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:22 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:09:22.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:22 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:09:22.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:22 smithi072 ceph-mon[30177]: osdmap e386: 8 total, 8 up, 8 in 2024-03-26T02:09:22.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:22 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:22.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:22 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:22.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:22 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-61"}]: dispatch 2024-03-26T02:09:22.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:22 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-61"}]: dispatch 2024-03-26T02:09:22.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:22 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:09:22.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:22 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:09:22.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:22 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:09:22.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:22 smithi072 ceph-mon[41619]: osdmap e386: 8 total, 8 up, 8 in 2024-03-26T02:09:22.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:22 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:22.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:22 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:22.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:22 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-61"}]: dispatch 2024-03-26T02:09:22.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:22 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-61"}]: dispatch 2024-03-26T02:09:22.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:22 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:09:22.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:22 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:09:22.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:22 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:09:22.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:22 smithi173 ceph-mon[33839]: osdmap e386: 8 total, 8 up, 8 in 2024-03-26T02:09:22.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:22 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:22.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:22 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:09:22.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:22 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-61"}]: dispatch 2024-03-26T02:09:22.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:22 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-61"}]: dispatch 2024-03-26T02:09:22.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:22 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:09:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:23 smithi072 ceph-mon[30177]: pgmap v544: 324 pgs: 324 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T02:09:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:23 smithi072 ceph-mon[30177]: osdmap e387: 8 total, 8 up, 8 in 2024-03-26T02:09:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:23 smithi072 ceph-mon[41619]: pgmap v544: 324 pgs: 324 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T02:09:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:23 smithi072 ceph-mon[41619]: osdmap e387: 8 total, 8 up, 8 in 2024-03-26T02:09:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:23 smithi173 ceph-mon[33839]: pgmap v544: 324 pgs: 324 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T02:09:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:23 smithi173 ceph-mon[33839]: osdmap e387: 8 total, 8 up, 8 in 2024-03-26T02:09:24.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:24 smithi072 ceph-mon[30177]: osdmap e388: 8 total, 8 up, 8 in 2024-03-26T02:09:24.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:24 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:24.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:24 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:24.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:24 smithi072 ceph-mon[41619]: osdmap e388: 8 total, 8 up, 8 in 2024-03-26T02:09:24.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:24 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:24.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:24 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:24.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:24 smithi173 ceph-mon[33839]: osdmap e388: 8 total, 8 up, 8 in 2024-03-26T02:09:24.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:24 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:24.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:24 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:09:25.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:09:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:09:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:09:25.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:25 smithi072 ceph-mon[30177]: pgmap v547: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T02:09:25.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:25 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:09:25.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:25 smithi072 ceph-mon[30177]: osdmap e389: 8 total, 8 up, 8 in 2024-03-26T02:09:25.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:09:25.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:09:25.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:25 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:09:25.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:25 smithi072 ceph-mon[41619]: pgmap v547: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T02:09:25.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:25 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:09:25.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:25 smithi072 ceph-mon[41619]: osdmap e389: 8 total, 8 up, 8 in 2024-03-26T02:09:25.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:09:25.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:09:25.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:25 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:09:25.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:25 smithi173 ceph-mon[33839]: pgmap v547: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T02:09:25.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:25 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:09:25.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:25 smithi173 ceph-mon[33839]: osdmap e389: 8 total, 8 up, 8 in 2024-03-26T02:09:25.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:09:25.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:09:25.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:25 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:09:26.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:26 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:09:26.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:26 smithi173 ceph-mon[33839]: osdmap e390: 8 total, 8 up, 8 in 2024-03-26T02:09:26.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:26 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:09:26.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:26 smithi072 ceph-mon[41619]: osdmap e390: 8 total, 8 up, 8 in 2024-03-26T02:09:26.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:26 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:09:26.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:26 smithi072 ceph-mon[30177]: osdmap e390: 8 total, 8 up, 8 in 2024-03-26T02:09:27.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:27 smithi173 ceph-mon[33839]: pgmap v550: 324 pgs: 324 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-26T02:09:27.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:27 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:09:27.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:27 smithi173 ceph-mon[33839]: osdmap e391: 8 total, 8 up, 8 in 2024-03-26T02:09:27.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:27 smithi072 ceph-mon[30177]: pgmap v550: 324 pgs: 324 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-26T02:09:27.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:27 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:09:27.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:27 smithi072 ceph-mon[30177]: osdmap e391: 8 total, 8 up, 8 in 2024-03-26T02:09:27.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:27 smithi072 ceph-mon[41619]: pgmap v550: 324 pgs: 324 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-26T02:09:27.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:27 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:09:27.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:27 smithi072 ceph-mon[41619]: osdmap e391: 8 total, 8 up, 8 in 2024-03-26T02:09:28.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:28 smithi173 ceph-mon[33839]: osdmap e392: 8 total, 8 up, 8 in 2024-03-26T02:09:28.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:28 smithi072 ceph-mon[30177]: osdmap e392: 8 total, 8 up, 8 in 2024-03-26T02:09:28.722 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:28 smithi072 ceph-mon[41619]: osdmap e392: 8 total, 8 up, 8 in 2024-03-26T02:09:29.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:29 smithi173 ceph-mon[33839]: pgmap v553: 324 pgs: 324 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-26T02:09:29.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:29 smithi173 ceph-mon[33839]: osdmap e393: 8 total, 8 up, 8 in 2024-03-26T02:09:29.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:29 smithi072 ceph-mon[30177]: pgmap v553: 324 pgs: 324 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-26T02:09:29.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:29 smithi072 ceph-mon[30177]: osdmap e393: 8 total, 8 up, 8 in 2024-03-26T02:09:29.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:29 smithi072 ceph-mon[41619]: pgmap v553: 324 pgs: 324 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-26T02:09:29.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:29 smithi072 ceph-mon[41619]: osdmap e393: 8 total, 8 up, 8 in 2024-03-26T02:09:30.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:30 smithi173 ceph-mon[33839]: osdmap e394: 8 total, 8 up, 8 in 2024-03-26T02:09:30.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:30 smithi072 ceph-mon[30177]: osdmap e394: 8 total, 8 up, 8 in 2024-03-26T02:09:30.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:30 smithi072 ceph-mon[41619]: osdmap e394: 8 total, 8 up, 8 in 2024-03-26T02:09:31.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:09:30 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:09:31.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:31 smithi173 ceph-mon[33839]: pgmap v556: 324 pgs: 324 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:09:31.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:31 smithi072 ceph-mon[41619]: pgmap v556: 324 pgs: 324 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:09:31.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:31 smithi072 ceph-mon[30177]: pgmap v556: 324 pgs: 324 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:09:32.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:32 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:09:32.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:32 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:09:32.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:32 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:09:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:33 smithi173 ceph-mon[33839]: pgmap v557: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-26T02:09:33.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:33 smithi072 ceph-mon[30177]: pgmap v557: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-26T02:09:33.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:33 smithi072 ceph-mon[41619]: pgmap v557: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-26T02:09:34.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:34 smithi173 ceph-mon[33839]: pgmap v558: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 967 B/s wr, 3 op/s 2024-03-26T02:09:34.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:34 smithi072 ceph-mon[30177]: pgmap v558: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 967 B/s wr, 3 op/s 2024-03-26T02:09:34.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:34 smithi072 ceph-mon[41619]: pgmap v558: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 967 B/s wr, 3 op/s 2024-03-26T02:09:35.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:09:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:09:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:09:36.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:36 smithi072 ceph-mon[30177]: pgmap v559: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-26T02:09:36.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:36 smithi072 ceph-mon[30177]: osdmap e395: 8 total, 8 up, 8 in 2024-03-26T02:09:36.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:36 smithi072 ceph-mon[41619]: pgmap v559: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-26T02:09:36.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:36 smithi072 ceph-mon[41619]: osdmap e395: 8 total, 8 up, 8 in 2024-03-26T02:09:37.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:36 smithi173 ceph-mon[33839]: pgmap v559: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-26T02:09:37.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:36 smithi173 ceph-mon[33839]: osdmap e395: 8 total, 8 up, 8 in 2024-03-26T02:09:37.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:37 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:09:37.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:37 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:09:38.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:37 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:09:38.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:38 smithi072 ceph-mon[30177]: pgmap v561: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 736 B/s wr, 3 op/s 2024-03-26T02:09:38.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:38 smithi072 ceph-mon[41619]: pgmap v561: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 736 B/s wr, 3 op/s 2024-03-26T02:09:39.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:38 smithi173 ceph-mon[33839]: pgmap v561: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 736 B/s wr, 3 op/s 2024-03-26T02:09:40.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:40 smithi072 ceph-mon[30177]: pgmap v562: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 614 B/s wr, 2 op/s 2024-03-26T02:09:40.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:40 smithi072 ceph-mon[30177]: osdmap e396: 8 total, 8 up, 8 in 2024-03-26T02:09:40.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:40 smithi072 ceph-mon[41619]: pgmap v562: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 614 B/s wr, 2 op/s 2024-03-26T02:09:40.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:40 smithi072 ceph-mon[41619]: osdmap e396: 8 total, 8 up, 8 in 2024-03-26T02:09:41.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:40 smithi173 ceph-mon[33839]: pgmap v562: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 614 B/s wr, 2 op/s 2024-03-26T02:09:41.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:40 smithi173 ceph-mon[33839]: osdmap e396: 8 total, 8 up, 8 in 2024-03-26T02:09:41.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:09:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:09:41.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:41 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:09:41.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:41 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:09:42.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:41 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:09:42.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:42 smithi072 ceph-mon[30177]: pgmap v564: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:09:42.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:42 smithi072 ceph-mon[41619]: pgmap v564: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:09:43.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:42 smithi173 ceph-mon[33839]: pgmap v564: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:09:43.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:43 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:43.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:43 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:43.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:43 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:43.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:43 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:09:43.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:43 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:43.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:43 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:43.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:43 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:43.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:43 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:09:44.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:43 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:09:44.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:43 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:09:44.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:43 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:09:44.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:43 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:09:44.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:44 smithi072 ceph-mon[30177]: pgmap v565: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:09:44.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:44 smithi072 ceph-mon[41619]: pgmap v565: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:09:45.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:44 smithi173 ceph-mon[33839]: pgmap v565: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:09:45.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:09:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:09:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:09:46.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:46 smithi072 ceph-mon[30177]: pgmap v566: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:09:46.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:46 smithi072 ceph-mon[30177]: osdmap e397: 8 total, 8 up, 8 in 2024-03-26T02:09:46.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:46 smithi072 ceph-mon[41619]: pgmap v566: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:09:46.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:46 smithi072 ceph-mon[41619]: osdmap e397: 8 total, 8 up, 8 in 2024-03-26T02:09:47.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:46 smithi173 ceph-mon[33839]: pgmap v566: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:09:47.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:46 smithi173 ceph-mon[33839]: osdmap e397: 8 total, 8 up, 8 in 2024-03-26T02:09:48.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:48 smithi072 ceph-mon[30177]: pgmap v568: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:09:48.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:48 smithi072 ceph-mon[41619]: pgmap v568: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:09:49.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:48 smithi173 ceph-mon[33839]: pgmap v568: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:09:50.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:50 smithi072 ceph-mon[30177]: pgmap v569: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:09:50.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:50 smithi072 ceph-mon[41619]: pgmap v569: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:09:51.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:09:50 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:09:51.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:50 smithi173 ceph-mon[33839]: pgmap v569: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:09:51.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:51 smithi072 ceph-mon[30177]: osdmap e398: 8 total, 8 up, 8 in 2024-03-26T02:09:51.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:51 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:09:51.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:51 smithi072 ceph-mon[41619]: osdmap e398: 8 total, 8 up, 8 in 2024-03-26T02:09:51.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:51 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:09:52.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:51 smithi173 ceph-mon[33839]: osdmap e398: 8 total, 8 up, 8 in 2024-03-26T02:09:52.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:51 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:09:52.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:52 smithi072 ceph-mon[30177]: pgmap v571: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T02:09:52.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:09:52.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:52 smithi072 ceph-mon[41619]: pgmap v571: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T02:09:52.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:09:53.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:52 smithi173 ceph-mon[33839]: pgmap v571: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T02:09:53.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:09:54.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:54 smithi072 ceph-mon[30177]: pgmap v572: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:09:54.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:54 smithi072 ceph-mon[41619]: pgmap v572: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:09:55.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:54 smithi173 ceph-mon[33839]: pgmap v572: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:09:55.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:09:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:09:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:09:56.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:56 smithi072 ceph-mon[30177]: pgmap v573: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:09:56.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:56 smithi072 ceph-mon[41619]: pgmap v573: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:09:57.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:56 smithi173 ceph-mon[33839]: pgmap v573: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:09:58.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:09:58 smithi072 ceph-mon[30177]: pgmap v574: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:09:58.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:09:58 smithi072 ceph-mon[41619]: pgmap v574: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:09:59.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:09:58 smithi173 ceph-mon[33839]: pgmap v574: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:10:00.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:00 smithi072 ceph-mon[30177]: pgmap v575: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:10:00.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:00 smithi072 ceph-mon[30177]: Health detail: HEALTH_WARN 4 pool(s) do not have an application enabled 2024-03-26T02:10:00.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:00 smithi072 ceph-mon[30177]: [WRN] POOL_APP_NOT_ENABLED: 4 pool(s) do not have an application enabled 2024-03-26T02:10:00.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:00 smithi072 ceph-mon[30177]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-26T02:10:00.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:00 smithi072 ceph-mon[30177]: application not enabled on pool 'SnapListsmithi072-83683-1' 2024-03-26T02:10:00.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:00 smithi072 ceph-mon[30177]: application not enabled on pool 'test_handler_errorsmithi072-83285-1' 2024-03-26T02:10:00.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:00 smithi072 ceph-mon[30177]: application not enabled on pool 'WatchNotifysmithi072-83710-1' 2024-03-26T02:10:00.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:00 smithi072 ceph-mon[30177]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-26T02:10:00.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:00 smithi072 ceph-mon[30177]: osdmap e399: 8 total, 8 up, 8 in 2024-03-26T02:10:00.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:00 smithi072 ceph-mon[41619]: pgmap v575: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:10:00.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:00 smithi072 ceph-mon[41619]: Health detail: HEALTH_WARN 4 pool(s) do not have an application enabled 2024-03-26T02:10:00.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:00 smithi072 ceph-mon[41619]: [WRN] POOL_APP_NOT_ENABLED: 4 pool(s) do not have an application enabled 2024-03-26T02:10:00.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:00 smithi072 ceph-mon[41619]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-26T02:10:00.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:00 smithi072 ceph-mon[41619]: application not enabled on pool 'SnapListsmithi072-83683-1' 2024-03-26T02:10:00.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:00 smithi072 ceph-mon[41619]: application not enabled on pool 'test_handler_errorsmithi072-83285-1' 2024-03-26T02:10:00.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:00 smithi072 ceph-mon[41619]: application not enabled on pool 'WatchNotifysmithi072-83710-1' 2024-03-26T02:10:00.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:00 smithi072 ceph-mon[41619]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-26T02:10:00.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:00 smithi072 ceph-mon[41619]: osdmap e399: 8 total, 8 up, 8 in 2024-03-26T02:10:01.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:10:00 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:10:01.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:00 smithi173 ceph-mon[33839]: pgmap v575: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T02:10:01.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:00 smithi173 ceph-mon[33839]: Health detail: HEALTH_WARN 4 pool(s) do not have an application enabled 2024-03-26T02:10:01.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:00 smithi173 ceph-mon[33839]: [WRN] POOL_APP_NOT_ENABLED: 4 pool(s) do not have an application enabled 2024-03-26T02:10:01.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:00 smithi173 ceph-mon[33839]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-26T02:10:01.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:00 smithi173 ceph-mon[33839]: application not enabled on pool 'SnapListsmithi072-83683-1' 2024-03-26T02:10:01.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:00 smithi173 ceph-mon[33839]: application not enabled on pool 'test_handler_errorsmithi072-83285-1' 2024-03-26T02:10:01.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:00 smithi173 ceph-mon[33839]: application not enabled on pool 'WatchNotifysmithi072-83710-1' 2024-03-26T02:10:01.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:00 smithi173 ceph-mon[33839]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-26T02:10:01.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:00 smithi173 ceph-mon[33839]: osdmap e399: 8 total, 8 up, 8 in 2024-03-26T02:10:02.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:02.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:01 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:02.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-63"}]: dispatch 2024-03-26T02:10:02.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:01 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-63"}]: dispatch 2024-03-26T02:10:02.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:01 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:10:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:01 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-63"}]: dispatch 2024-03-26T02:10:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:01 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-63"}]: dispatch 2024-03-26T02:10:02.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:01 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:10:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:01 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-63"}]: dispatch 2024-03-26T02:10:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:01 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-63"}]: dispatch 2024-03-26T02:10:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:01 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:10:03.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:02 smithi173 ceph-mon[33839]: pgmap v577: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:10:03.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:02 smithi173 ceph-mon[33839]: osdmap e400: 8 total, 8 up, 8 in 2024-03-26T02:10:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:02 smithi072 ceph-mon[30177]: pgmap v577: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:10:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:02 smithi072 ceph-mon[30177]: osdmap e400: 8 total, 8 up, 8 in 2024-03-26T02:10:03.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:02 smithi072 ceph-mon[41619]: pgmap v577: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:10:03.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:02 smithi072 ceph-mon[41619]: osdmap e400: 8 total, 8 up, 8 in 2024-03-26T02:10:04.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:03 smithi173 ceph-mon[33839]: osdmap e401: 8 total, 8 up, 8 in 2024-03-26T02:10:04.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:04.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:03 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:04.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:03 smithi072 ceph-mon[30177]: osdmap e401: 8 total, 8 up, 8 in 2024-03-26T02:10:04.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:04.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:03 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:04.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:03 smithi072 ceph-mon[41619]: osdmap e401: 8 total, 8 up, 8 in 2024-03-26T02:10:04.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:04.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:03 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:05.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:04 smithi173 ceph-mon[33839]: pgmap v580: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:10:05.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:04 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:10:05.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:04 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:10:05.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:04 smithi173 ceph-mon[33839]: osdmap e402: 8 total, 8 up, 8 in 2024-03-26T02:10:05.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:04 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:10:05.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:04 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:10:05.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:04 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:10:05.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:10:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:10:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:10:05.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:04 smithi072 ceph-mon[30177]: pgmap v580: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:10:05.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:04 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:10:05.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:04 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:10:05.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:04 smithi072 ceph-mon[30177]: osdmap e402: 8 total, 8 up, 8 in 2024-03-26T02:10:05.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:04 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:10:05.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:04 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:10:05.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:04 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:10:05.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:04 smithi072 ceph-mon[41619]: pgmap v580: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:10:05.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:04 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:10:05.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:04 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:10:05.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:04 smithi072 ceph-mon[41619]: osdmap e402: 8 total, 8 up, 8 in 2024-03-26T02:10:05.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:04 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:10:05.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:04 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:10:05.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:04 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:10:06.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:10:06.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:05 smithi173 ceph-mon[33839]: osdmap e403: 8 total, 8 up, 8 in 2024-03-26T02:10:06.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:10:06.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:05 smithi072 ceph-mon[30177]: osdmap e403: 8 total, 8 up, 8 in 2024-03-26T02:10:06.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:10:06.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:05 smithi072 ceph-mon[41619]: osdmap e403: 8 total, 8 up, 8 in 2024-03-26T02:10:07.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:06 smithi173 ceph-mon[33839]: pgmap v583: 324 pgs: 324 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:10:07.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:06 smithi173 ceph-mon[33839]: osdmap e404: 8 total, 8 up, 8 in 2024-03-26T02:10:07.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:06 smithi072 ceph-mon[30177]: pgmap v583: 324 pgs: 324 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:10:07.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:06 smithi072 ceph-mon[30177]: osdmap e404: 8 total, 8 up, 8 in 2024-03-26T02:10:07.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:06 smithi072 ceph-mon[41619]: pgmap v583: 324 pgs: 324 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:10:07.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:06 smithi072 ceph-mon[41619]: osdmap e404: 8 total, 8 up, 8 in 2024-03-26T02:10:08.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:07 smithi173 ceph-mon[33839]: osdmap e405: 8 total, 8 up, 8 in 2024-03-26T02:10:08.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:07 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:10:08.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:07 smithi072 ceph-mon[30177]: osdmap e405: 8 total, 8 up, 8 in 2024-03-26T02:10:08.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:07 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:10:08.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:07 smithi072 ceph-mon[41619]: osdmap e405: 8 total, 8 up, 8 in 2024-03-26T02:10:08.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:07 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:10:09.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:08 smithi173 ceph-mon[33839]: pgmap v586: 324 pgs: 324 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-26T02:10:09.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:08 smithi173 ceph-mon[33839]: osdmap e406: 8 total, 8 up, 8 in 2024-03-26T02:10:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:08 smithi072 ceph-mon[30177]: pgmap v586: 324 pgs: 324 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-26T02:10:09.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:08 smithi072 ceph-mon[30177]: osdmap e406: 8 total, 8 up, 8 in 2024-03-26T02:10:09.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:08 smithi072 ceph-mon[41619]: pgmap v586: 324 pgs: 324 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-26T02:10:09.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:08 smithi072 ceph-mon[41619]: osdmap e406: 8 total, 8 up, 8 in 2024-03-26T02:10:11.021 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:10 smithi072 ceph-mon[30177]: pgmap v588: 324 pgs: 324 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T02:10:11.022 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:10 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:10:11.022 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:10 smithi072 ceph-mon[41619]: pgmap v588: 324 pgs: 324 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T02:10:11.022 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:10 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:10:11.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:10 smithi173 ceph-mon[33839]: pgmap v588: 324 pgs: 324 active+clean; 8.4 MiB data, 647 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T02:10:11.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:10 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:10:11.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:10:10 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:10:12.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:11 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:10:12.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:11 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:10:12.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:11 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:10:13.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:12 smithi173 ceph-mon[33839]: pgmap v589: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 682 B/s wr, 4 op/s 2024-03-26T02:10:13.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:12 smithi072 ceph-mon[30177]: pgmap v589: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 682 B/s wr, 4 op/s 2024-03-26T02:10:13.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:12 smithi072 ceph-mon[41619]: pgmap v589: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 682 B/s wr, 4 op/s 2024-03-26T02:10:15.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:14 smithi173 ceph-mon[33839]: pgmap v590: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 521 B/s wr, 3 op/s 2024-03-26T02:10:15.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:14 smithi072 ceph-mon[30177]: pgmap v590: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 521 B/s wr, 3 op/s 2024-03-26T02:10:15.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:14 smithi072 ceph-mon[41619]: pgmap v590: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 521 B/s wr, 3 op/s 2024-03-26T02:10:15.220 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:10:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:10:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:10:16.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:16 smithi072 ceph-mon[30177]: pgmap v591: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 462 B/s wr, 3 op/s 2024-03-26T02:10:16.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:16 smithi072 ceph-mon[30177]: osdmap e407: 8 total, 8 up, 8 in 2024-03-26T02:10:16.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:16 smithi072 ceph-mon[41619]: pgmap v591: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 462 B/s wr, 3 op/s 2024-03-26T02:10:16.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:16 smithi072 ceph-mon[41619]: osdmap e407: 8 total, 8 up, 8 in 2024-03-26T02:10:17.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:16 smithi173 ceph-mon[33839]: pgmap v591: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 462 B/s wr, 3 op/s 2024-03-26T02:10:17.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:16 smithi173 ceph-mon[33839]: osdmap e407: 8 total, 8 up, 8 in 2024-03-26T02:10:18.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:18 smithi072 ceph-mon[30177]: pgmap v593: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 416 B/s wr, 3 op/s 2024-03-26T02:10:18.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:18.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:18.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-65"}]: dispatch 2024-03-26T02:10:18.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-65"}]: dispatch 2024-03-26T02:10:18.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:18 smithi072 ceph-mon[41619]: pgmap v593: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 416 B/s wr, 3 op/s 2024-03-26T02:10:18.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:18.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:18.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-65"}]: dispatch 2024-03-26T02:10:18.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-65"}]: dispatch 2024-03-26T02:10:19.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:18 smithi173 ceph-mon[33839]: pgmap v593: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 416 B/s wr, 3 op/s 2024-03-26T02:10:19.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:19.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:19.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-65"}]: dispatch 2024-03-26T02:10:19.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-65"}]: dispatch 2024-03-26T02:10:19.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:19 smithi072 ceph-mon[30177]: osdmap e408: 8 total, 8 up, 8 in 2024-03-26T02:10:19.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:19 smithi072 ceph-mon[41619]: osdmap e408: 8 total, 8 up, 8 in 2024-03-26T02:10:20.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:19 smithi173 ceph-mon[33839]: osdmap e408: 8 total, 8 up, 8 in 2024-03-26T02:10:20.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:20 smithi072 ceph-mon[30177]: pgmap v595: 292 pgs: 292 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:10:20.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:20 smithi072 ceph-mon[30177]: osdmap e409: 8 total, 8 up, 8 in 2024-03-26T02:10:20.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:20.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:20 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:20.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:20 smithi072 ceph-mon[41619]: pgmap v595: 292 pgs: 292 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:10:20.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:20 smithi072 ceph-mon[41619]: osdmap e409: 8 total, 8 up, 8 in 2024-03-26T02:10:20.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:20.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:20 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:21.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:10:20 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:10:21.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:20 smithi173 ceph-mon[33839]: pgmap v595: 292 pgs: 292 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:10:21.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:20 smithi173 ceph-mon[33839]: osdmap e409: 8 total, 8 up, 8 in 2024-03-26T02:10:21.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:21.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:20 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:21.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:21 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:10:21.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:21 smithi072 ceph-mon[41619]: osdmap e410: 8 total, 8 up, 8 in 2024-03-26T02:10:21.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:10:21.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:21 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:10:21.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:21 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:10:21.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:21 smithi072 ceph-mon[30177]: osdmap e410: 8 total, 8 up, 8 in 2024-03-26T02:10:21.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:10:21.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:21 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:10:22.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:21 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:10:22.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:21 smithi173 ceph-mon[33839]: osdmap e410: 8 total, 8 up, 8 in 2024-03-26T02:10:22.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:10:22.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:21 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:10:23.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:22 smithi072 ceph-mon[30177]: pgmap v598: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:10:23.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:22 smithi072 ceph-mon[30177]: osdmap e411: 8 total, 8 up, 8 in 2024-03-26T02:10:23.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:22 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:23.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:22 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:23.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:22 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-67"}]: dispatch 2024-03-26T02:10:23.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:22 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-67"}]: dispatch 2024-03-26T02:10:23.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:22 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:10:23.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:22 smithi072 ceph-mon[41619]: pgmap v598: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:10:23.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:22 smithi072 ceph-mon[41619]: osdmap e411: 8 total, 8 up, 8 in 2024-03-26T02:10:23.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:22 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:23.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:22 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:23.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:22 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-67"}]: dispatch 2024-03-26T02:10:23.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:22 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-67"}]: dispatch 2024-03-26T02:10:23.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:22 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:10:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:22 smithi173 ceph-mon[33839]: pgmap v598: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:10:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:22 smithi173 ceph-mon[33839]: osdmap e411: 8 total, 8 up, 8 in 2024-03-26T02:10:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:22 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:23.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:22 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:23.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:22 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-67"}]: dispatch 2024-03-26T02:10:23.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:22 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-67"}]: dispatch 2024-03-26T02:10:23.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:22 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:10:24.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:23 smithi173 ceph-mon[33839]: osdmap e412: 8 total, 8 up, 8 in 2024-03-26T02:10:24.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:23 smithi173 ceph-mon[33839]: osdmap e413: 8 total, 8 up, 8 in 2024-03-26T02:10:24.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:23 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:24.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:23 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:24.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:23 smithi072 ceph-mon[30177]: osdmap e412: 8 total, 8 up, 8 in 2024-03-26T02:10:24.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:23 smithi072 ceph-mon[30177]: osdmap e413: 8 total, 8 up, 8 in 2024-03-26T02:10:24.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:23 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:24.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:23 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:24.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:23 smithi072 ceph-mon[41619]: osdmap e412: 8 total, 8 up, 8 in 2024-03-26T02:10:24.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:23 smithi072 ceph-mon[41619]: osdmap e413: 8 total, 8 up, 8 in 2024-03-26T02:10:24.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:23 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:24.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:23 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:24 smithi173 ceph-mon[33839]: pgmap v601: 292 pgs: 292 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:10:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:24 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:10:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:24 smithi173 ceph-mon[33839]: osdmap e414: 8 total, 8 up, 8 in 2024-03-26T02:10:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:24 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:10:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:24 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:24 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:24 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-69"}]: dispatch 2024-03-26T02:10:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:24 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-69"}]: dispatch 2024-03-26T02:10:25.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:10:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:10:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:10:25.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:24 smithi072 ceph-mon[30177]: pgmap v601: 292 pgs: 292 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:10:25.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:24 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:10:25.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:24 smithi072 ceph-mon[30177]: osdmap e414: 8 total, 8 up, 8 in 2024-03-26T02:10:25.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:24 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:10:25.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:24 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:25.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:24 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:25.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:24 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-69"}]: dispatch 2024-03-26T02:10:25.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:24 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-69"}]: dispatch 2024-03-26T02:10:25.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:24 smithi072 ceph-mon[41619]: pgmap v601: 292 pgs: 292 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:10:25.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:24 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:10:25.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:24 smithi072 ceph-mon[41619]: osdmap e414: 8 total, 8 up, 8 in 2024-03-26T02:10:25.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:24 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:10:25.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:24 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:25.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:24 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:25.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:24 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-69"}]: dispatch 2024-03-26T02:10:25.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:24 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-69"}]: dispatch 2024-03-26T02:10:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:26 smithi173 ceph-mon[33839]: pgmap v604: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T02:10:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:26 smithi173 ceph-mon[33839]: osdmap e415: 8 total, 8 up, 8 in 2024-03-26T02:10:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:26 smithi072 ceph-mon[30177]: pgmap v604: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T02:10:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:26 smithi072 ceph-mon[30177]: osdmap e415: 8 total, 8 up, 8 in 2024-03-26T02:10:27.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:26 smithi072 ceph-mon[41619]: pgmap v604: 324 pgs: 324 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T02:10:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:26 smithi072 ceph-mon[41619]: osdmap e415: 8 total, 8 up, 8 in 2024-03-26T02:10:28.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:27 smithi173 ceph-mon[33839]: osdmap e416: 8 total, 8 up, 8 in 2024-03-26T02:10:28.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:27 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:28.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:27 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:28.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:27 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:10:28.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:27 smithi173 ceph-mon[33839]: osdmap e417: 8 total, 8 up, 8 in 2024-03-26T02:10:28.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:27 smithi072 ceph-mon[41619]: osdmap e416: 8 total, 8 up, 8 in 2024-03-26T02:10:28.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:27 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:28.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:27 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:28.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:27 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:10:28.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:27 smithi072 ceph-mon[41619]: osdmap e417: 8 total, 8 up, 8 in 2024-03-26T02:10:28.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:27 smithi072 ceph-mon[30177]: osdmap e416: 8 total, 8 up, 8 in 2024-03-26T02:10:28.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:27 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:28.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:27 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:28.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:27 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:10:28.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:27 smithi072 ceph-mon[30177]: osdmap e417: 8 total, 8 up, 8 in 2024-03-26T02:10:29.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:28 smithi173 ceph-mon[33839]: pgmap v607: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T02:10:29.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:28 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:10:29.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:28 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:10:29.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:28 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:10:29.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:28 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:10:29.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:28 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:10:29.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:28 smithi173 ceph-mon[33839]: osdmap e418: 8 total, 8 up, 8 in 2024-03-26T02:10:29.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:28 smithi072 ceph-mon[30177]: pgmap v607: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T02:10:29.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:28 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:10:29.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:28 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:10:29.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:28 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:10:29.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:28 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:10:29.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:28 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:10:29.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:28 smithi072 ceph-mon[30177]: osdmap e418: 8 total, 8 up, 8 in 2024-03-26T02:10:29.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:28 smithi072 ceph-mon[41619]: pgmap v607: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T02:10:29.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:28 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:10:29.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:28 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:10:29.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:28 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:10:29.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:28 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:10:29.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:28 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:10:29.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:28 smithi072 ceph-mon[41619]: osdmap e418: 8 total, 8 up, 8 in 2024-03-26T02:10:31.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:30 smithi173 ceph-mon[33839]: pgmap v610: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:10:31.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:30 smithi173 ceph-mon[33839]: osdmap e419: 8 total, 8 up, 8 in 2024-03-26T02:10:31.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:10:30 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:10:31.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:30 smithi072 ceph-mon[30177]: pgmap v610: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:10:31.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:30 smithi072 ceph-mon[30177]: osdmap e419: 8 total, 8 up, 8 in 2024-03-26T02:10:31.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:30 smithi072 ceph-mon[41619]: pgmap v610: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:10:31.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:30 smithi072 ceph-mon[41619]: osdmap e419: 8 total, 8 up, 8 in 2024-03-26T02:10:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:31 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:10:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:31 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:10:32.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:31 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:10:33.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:32 smithi173 ceph-mon[33839]: pgmap v612: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-03-26T02:10:33.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:32 smithi072 ceph-mon[30177]: pgmap v612: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-03-26T02:10:33.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:32 smithi072 ceph-mon[41619]: pgmap v612: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-03-26T02:10:35.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:34 smithi173 ceph-mon[33839]: pgmap v613: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-26T02:10:35.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:34 smithi072 ceph-mon[30177]: pgmap v613: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-26T02:10:35.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:10:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:10:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:10:35.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:34 smithi072 ceph-mon[41619]: pgmap v613: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-26T02:10:36.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:35 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:10:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:35 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:10:36.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:35 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:10:37.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:36 smithi173 ceph-mon[33839]: pgmap v614: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-26T02:10:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:36 smithi072 ceph-mon[30177]: pgmap v614: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-26T02:10:37.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:36 smithi072 ceph-mon[41619]: pgmap v614: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-03-26T02:10:38.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:37 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:10:38.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:37 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:10:38.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:37 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:10:39.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:38 smithi173 ceph-mon[33839]: pgmap v615: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T02:10:39.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:38 smithi072 ceph-mon[30177]: pgmap v615: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T02:10:39.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:38 smithi072 ceph-mon[41619]: pgmap v615: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T02:10:41.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:10:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:10:41.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:40 smithi173 ceph-mon[33839]: pgmap v616: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-26T02:10:41.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:40 smithi072 ceph-mon[30177]: pgmap v616: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-26T02:10:41.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:40 smithi072 ceph-mon[41619]: pgmap v616: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-26T02:10:42.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:41 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:10:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:41 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:10:42.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:41 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:10:43.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:42 smithi173 ceph-mon[33839]: pgmap v617: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-26T02:10:43.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:42 smithi072 ceph-mon[30177]: pgmap v617: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-26T02:10:43.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:42 smithi072 ceph-mon[41619]: pgmap v617: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-26T02:10:44.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:43 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:10:44.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:43 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:10:44.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:43 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:10:45.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:44 smithi173 ceph-mon[33839]: pgmap v618: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T02:10:45.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:44 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:10:45.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:44 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:10:45.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:44 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:10:45.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:10:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:10:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:10:45.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:44 smithi072 ceph-mon[30177]: pgmap v618: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T02:10:45.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:44 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:10:45.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:44 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:10:45.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:44 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:10:45.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:44 smithi072 ceph-mon[41619]: pgmap v618: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T02:10:45.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:44 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:10:45.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:44 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:10:45.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:44 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:10:47.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:46 smithi173 ceph-mon[33839]: pgmap v619: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T02:10:47.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:46 smithi072 ceph-mon[41619]: pgmap v619: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T02:10:47.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:46 smithi072 ceph-mon[30177]: pgmap v619: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T02:10:49.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:48 smithi072 ceph-mon[30177]: pgmap v620: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T02:10:49.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:48 smithi072 ceph-mon[41619]: pgmap v620: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T02:10:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:48 smithi173 ceph-mon[33839]: pgmap v620: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T02:10:51.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:50 smithi072 ceph-mon[41619]: pgmap v621: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T02:10:51.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:50 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:51.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:50 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:51.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:50 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-71"}]: dispatch 2024-03-26T02:10:51.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:50 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-71"}]: dispatch 2024-03-26T02:10:51.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:50 smithi072 ceph-mon[30177]: pgmap v621: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T02:10:51.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:50 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:51.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:50 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:51.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:50 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-71"}]: dispatch 2024-03-26T02:10:51.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:50 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-71"}]: dispatch 2024-03-26T02:10:51.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:10:50 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:10:51.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:50 smithi173 ceph-mon[33839]: pgmap v621: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T02:10:51.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:50 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:51.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:50 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:10:51.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:50 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-71"}]: dispatch 2024-03-26T02:10:51.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:50 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-71"}]: dispatch 2024-03-26T02:10:52.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:51 smithi072 ceph-mon[41619]: osdmap e420: 8 total, 8 up, 8 in 2024-03-26T02:10:52.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:51 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:10:52.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:51 smithi072 ceph-mon[30177]: osdmap e420: 8 total, 8 up, 8 in 2024-03-26T02:10:52.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:51 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:10:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:51 smithi173 ceph-mon[33839]: osdmap e420: 8 total, 8 up, 8 in 2024-03-26T02:10:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:51 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:10:53.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:52 smithi072 ceph-mon[30177]: pgmap v623: 292 pgs: 292 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:10:53.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:52 smithi072 ceph-mon[30177]: osdmap e421: 8 total, 8 up, 8 in 2024-03-26T02:10:53.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:52 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:53.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:52 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:53.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:10:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:52 smithi072 ceph-mon[41619]: pgmap v623: 292 pgs: 292 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:10:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:52 smithi072 ceph-mon[41619]: osdmap e421: 8 total, 8 up, 8 in 2024-03-26T02:10:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:52 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:52 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:10:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:52 smithi173 ceph-mon[33839]: pgmap v623: 292 pgs: 292 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:10:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:52 smithi173 ceph-mon[33839]: osdmap e421: 8 total, 8 up, 8 in 2024-03-26T02:10:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:52 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:52 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:10:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:10:54.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:53 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:10:54.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:53 smithi072 ceph-mon[30177]: osdmap e422: 8 total, 8 up, 8 in 2024-03-26T02:10:54.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:53 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:10:54.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:53 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:10:54.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:53 smithi072 ceph-mon[41619]: osdmap e422: 8 total, 8 up, 8 in 2024-03-26T02:10:54.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:53 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:10:54.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:53 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:10:54.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:53 smithi173 ceph-mon[33839]: osdmap e422: 8 total, 8 up, 8 in 2024-03-26T02:10:54.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:53 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:10:55.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:54 smithi072 ceph-mon[30177]: pgmap v626: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:10:55.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:54 smithi072 ceph-mon[30177]: osdmap e423: 8 total, 8 up, 8 in 2024-03-26T02:10:55.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:10:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:10:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:10:55.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:54 smithi072 ceph-mon[41619]: pgmap v626: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:10:55.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:54 smithi072 ceph-mon[41619]: osdmap e423: 8 total, 8 up, 8 in 2024-03-26T02:10:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:54 smithi173 ceph-mon[33839]: pgmap v626: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:10:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:54 smithi173 ceph-mon[33839]: osdmap e423: 8 total, 8 up, 8 in 2024-03-26T02:10:56.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:55 smithi072 ceph-mon[30177]: osdmap e424: 8 total, 8 up, 8 in 2024-03-26T02:10:56.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:55 smithi072 ceph-mon[41619]: osdmap e424: 8 total, 8 up, 8 in 2024-03-26T02:10:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:55 smithi173 ceph-mon[33839]: osdmap e424: 8 total, 8 up, 8 in 2024-03-26T02:10:57.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:56 smithi072 ceph-mon[30177]: pgmap v629: 324 pgs: 324 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-26T02:10:57.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:56 smithi072 ceph-mon[41619]: pgmap v629: 324 pgs: 324 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-26T02:10:57.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:56 smithi173 ceph-mon[33839]: pgmap v629: 324 pgs: 324 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-26T02:10:59.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:10:58 smithi072 ceph-mon[30177]: pgmap v630: 324 pgs: 324 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-26T02:10:59.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:10:58 smithi072 ceph-mon[41619]: pgmap v630: 324 pgs: 324 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-26T02:10:59.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:10:58 smithi173 ceph-mon[33839]: pgmap v630: 324 pgs: 324 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-26T02:11:01.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:00 smithi072 ceph-mon[30177]: pgmap v631: 324 pgs: 324 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-03-26T02:11:01.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:00 smithi072 ceph-mon[41619]: pgmap v631: 324 pgs: 324 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-03-26T02:11:01.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:00 smithi173 ceph-mon[33839]: pgmap v631: 324 pgs: 324 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.0 KiB/s wr, 3 op/s 2024-03-26T02:11:01.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:11:00 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:11:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:01 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:11:02.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:01 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:11:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:01 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:11:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:02 smithi072 ceph-mon[30177]: pgmap v632: 324 pgs: 324 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 895 B/s wr, 3 op/s 2024-03-26T02:11:03.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:02 smithi072 ceph-mon[41619]: pgmap v632: 324 pgs: 324 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 895 B/s wr, 3 op/s 2024-03-26T02:11:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:02 smithi173 ceph-mon[33839]: pgmap v632: 324 pgs: 324 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 895 B/s wr, 3 op/s 2024-03-26T02:11:05.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:04 smithi072 ceph-mon[30177]: pgmap v633: 324 pgs: 324 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 734 B/s wr, 3 op/s 2024-03-26T02:11:05.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:11:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:11:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:11:05.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:04 smithi072 ceph-mon[41619]: pgmap v633: 324 pgs: 324 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 734 B/s wr, 3 op/s 2024-03-26T02:11:05.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:04 smithi173 ceph-mon[33839]: pgmap v633: 324 pgs: 324 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 734 B/s wr, 3 op/s 2024-03-26T02:11:06.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:06.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:06.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-73"}]: dispatch 2024-03-26T02:11:06.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-73"}]: dispatch 2024-03-26T02:11:06.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:06.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:06.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-73"}]: dispatch 2024-03-26T02:11:06.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-73"}]: dispatch 2024-03-26T02:11:06.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:06.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:06.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-73"}]: dispatch 2024-03-26T02:11:06.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-73"}]: dispatch 2024-03-26T02:11:07.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:06 smithi072 ceph-mon[30177]: pgmap v634: 324 pgs: 324 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 665 B/s wr, 3 op/s 2024-03-26T02:11:07.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:06 smithi072 ceph-mon[30177]: osdmap e425: 8 total, 8 up, 8 in 2024-03-26T02:11:07.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:06 smithi072 ceph-mon[41619]: pgmap v634: 324 pgs: 324 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 665 B/s wr, 3 op/s 2024-03-26T02:11:07.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:06 smithi072 ceph-mon[41619]: osdmap e425: 8 total, 8 up, 8 in 2024-03-26T02:11:07.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:06 smithi173 ceph-mon[33839]: pgmap v634: 324 pgs: 324 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 665 B/s wr, 3 op/s 2024-03-26T02:11:07.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:06 smithi173 ceph-mon[33839]: osdmap e425: 8 total, 8 up, 8 in 2024-03-26T02:11:08.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:07 smithi072 ceph-mon[41619]: osdmap e426: 8 total, 8 up, 8 in 2024-03-26T02:11:08.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:08.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:07 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:08.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:07 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:11:08.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:07 smithi072 ceph-mon[30177]: osdmap e426: 8 total, 8 up, 8 in 2024-03-26T02:11:08.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:08.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:07 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:08.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:07 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:11:08.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:07 smithi173 ceph-mon[33839]: osdmap e426: 8 total, 8 up, 8 in 2024-03-26T02:11:08.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:08.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:07 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:08.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:07 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:11:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:08 smithi072 ceph-mon[30177]: pgmap v637: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:11:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:08 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:11:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:11:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:08 smithi072 ceph-mon[30177]: osdmap e427: 8 total, 8 up, 8 in 2024-03-26T02:11:09.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:11:09.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:08 smithi072 ceph-mon[41619]: pgmap v637: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:11:09.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:08 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:11:09.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:11:09.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:08 smithi072 ceph-mon[41619]: osdmap e427: 8 total, 8 up, 8 in 2024-03-26T02:11:09.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:11:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:08 smithi173 ceph-mon[33839]: pgmap v637: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:11:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:08 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:11:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:11:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:08 smithi173 ceph-mon[33839]: osdmap e427: 8 total, 8 up, 8 in 2024-03-26T02:11:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:11:10.214 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:09 smithi173 ceph-mon[33839]: osdmap e428: 8 total, 8 up, 8 in 2024-03-26T02:11:10.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:09 smithi072 ceph-mon[30177]: osdmap e428: 8 total, 8 up, 8 in 2024-03-26T02:11:10.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:09 smithi072 ceph-mon[41619]: osdmap e428: 8 total, 8 up, 8 in 2024-03-26T02:11:11.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:10 smithi072 ceph-mon[30177]: pgmap v640: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:11:11.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:10 smithi072 ceph-mon[30177]: osdmap e429: 8 total, 8 up, 8 in 2024-03-26T02:11:11.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:11.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:11.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-75"}]: dispatch 2024-03-26T02:11:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-75"}]: dispatch 2024-03-26T02:11:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:10 smithi072 ceph-mon[41619]: pgmap v640: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:11:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:10 smithi072 ceph-mon[41619]: osdmap e429: 8 total, 8 up, 8 in 2024-03-26T02:11:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-75"}]: dispatch 2024-03-26T02:11:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-75"}]: dispatch 2024-03-26T02:11:11.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:11:10 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:11:11.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:10 smithi173 ceph-mon[33839]: pgmap v640: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 662 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:11:11.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:10 smithi173 ceph-mon[33839]: osdmap e429: 8 total, 8 up, 8 in 2024-03-26T02:11:11.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:11.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:11.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-75"}]: dispatch 2024-03-26T02:11:11.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-75"}]: dispatch 2024-03-26T02:11:12.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:11 smithi072 ceph-mon[30177]: osdmap e430: 8 total, 8 up, 8 in 2024-03-26T02:11:12.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:11 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:11:12.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:11 smithi072 ceph-mon[41619]: osdmap e430: 8 total, 8 up, 8 in 2024-03-26T02:11:12.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:11 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:11:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:11 smithi173 ceph-mon[33839]: osdmap e430: 8 total, 8 up, 8 in 2024-03-26T02:11:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:11 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:11:13.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:12 smithi072 ceph-mon[30177]: pgmap v643: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T02:11:13.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:12 smithi072 ceph-mon[30177]: osdmap e431: 8 total, 8 up, 8 in 2024-03-26T02:11:13.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:13.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:13.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:12 smithi072 ceph-mon[41619]: pgmap v643: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T02:11:13.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:12 smithi072 ceph-mon[41619]: osdmap e431: 8 total, 8 up, 8 in 2024-03-26T02:11:13.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:13.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:12 smithi173 ceph-mon[33839]: pgmap v643: 292 pgs: 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T02:11:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:12 smithi173 ceph-mon[33839]: osdmap e431: 8 total, 8 up, 8 in 2024-03-26T02:11:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:14.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:13 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:11:14.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:13 smithi072 ceph-mon[41619]: osdmap e432: 8 total, 8 up, 8 in 2024-03-26T02:11:14.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:13 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:11:14.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:13 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:11:14.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:13 smithi072 ceph-mon[30177]: osdmap e432: 8 total, 8 up, 8 in 2024-03-26T02:11:14.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:13 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:11:14.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:13 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:11:14.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:13 smithi173 ceph-mon[33839]: osdmap e432: 8 total, 8 up, 8 in 2024-03-26T02:11:14.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:13 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:11:15.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:11:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:11:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:11:15.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:15 smithi072 ceph-mon[30177]: pgmap v646: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T02:11:15.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:15 smithi072 ceph-mon[30177]: osdmap e433: 8 total, 8 up, 8 in 2024-03-26T02:11:15.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:15.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:15.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-77"}]: dispatch 2024-03-26T02:11:15.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-77"}]: dispatch 2024-03-26T02:11:15.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:15 smithi072 ceph-mon[41619]: pgmap v646: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T02:11:15.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:15 smithi072 ceph-mon[41619]: osdmap e433: 8 total, 8 up, 8 in 2024-03-26T02:11:15.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:15.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:15.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-77"}]: dispatch 2024-03-26T02:11:15.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-77"}]: dispatch 2024-03-26T02:11:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:15 smithi173 ceph-mon[33839]: pgmap v646: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 663 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T02:11:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:15 smithi173 ceph-mon[33839]: osdmap e433: 8 total, 8 up, 8 in 2024-03-26T02:11:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-77"}]: dispatch 2024-03-26T02:11:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-77"}]: dispatch 2024-03-26T02:11:16.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:16 smithi173 ceph-mon[33839]: osdmap e434: 8 total, 8 up, 8 in 2024-03-26T02:11:16.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:16 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:11:16.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:16 smithi173 ceph-mon[33839]: osdmap e435: 8 total, 8 up, 8 in 2024-03-26T02:11:16.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:16 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:16.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:16 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:16.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:16 smithi072 ceph-mon[30177]: osdmap e434: 8 total, 8 up, 8 in 2024-03-26T02:11:16.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:16 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:11:16.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:16 smithi072 ceph-mon[30177]: osdmap e435: 8 total, 8 up, 8 in 2024-03-26T02:11:16.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:16 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:16.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:16 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:16.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:16 smithi072 ceph-mon[41619]: osdmap e434: 8 total, 8 up, 8 in 2024-03-26T02:11:16.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:16 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:11:16.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:16 smithi072 ceph-mon[41619]: osdmap e435: 8 total, 8 up, 8 in 2024-03-26T02:11:16.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:16 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:16.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:16 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:17.566 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:17 smithi173 ceph-mon[33839]: pgmap v649: 292 pgs: 292 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 511 B/s wr, 22 op/s 2024-03-26T02:11:17.566 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:17 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:11:17.566 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:17 smithi173 ceph-mon[33839]: osdmap e436: 8 total, 8 up, 8 in 2024-03-26T02:11:17.566 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:17 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:11:17.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:17 smithi072 ceph-mon[30177]: pgmap v649: 292 pgs: 292 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 511 B/s wr, 22 op/s 2024-03-26T02:11:17.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:17 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:11:17.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:17 smithi072 ceph-mon[30177]: osdmap e436: 8 total, 8 up, 8 in 2024-03-26T02:11:17.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:17 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:11:17.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:17 smithi072 ceph-mon[41619]: pgmap v649: 292 pgs: 292 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 511 B/s wr, 22 op/s 2024-03-26T02:11:17.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:17 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:11:17.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:17 smithi072 ceph-mon[41619]: osdmap e436: 8 total, 8 up, 8 in 2024-03-26T02:11:17.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:17 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:11:19.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:18 smithi173 ceph-mon[33839]: pgmap v652: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 511 B/s wr, 22 op/s 2024-03-26T02:11:19.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:18 smithi173 ceph-mon[33839]: osdmap e437: 8 total, 8 up, 8 in 2024-03-26T02:11:19.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:18 smithi072 ceph-mon[30177]: pgmap v652: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 511 B/s wr, 22 op/s 2024-03-26T02:11:19.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:18 smithi072 ceph-mon[30177]: osdmap e437: 8 total, 8 up, 8 in 2024-03-26T02:11:19.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:18 smithi072 ceph-mon[41619]: pgmap v652: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 511 B/s wr, 22 op/s 2024-03-26T02:11:19.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:18 smithi072 ceph-mon[41619]: osdmap e437: 8 total, 8 up, 8 in 2024-03-26T02:11:20.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:19 smithi173 ceph-mon[33839]: osdmap e438: 8 total, 8 up, 8 in 2024-03-26T02:11:20.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "294.1f"}]: dispatch 2024-03-26T02:11:20.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:19 smithi173 ceph-mon[33839]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "294.1f"}]: dispatch 2024-03-26T02:11:20.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:19 smithi173 ceph-mon[33839]: 294.1f deep-scrub starts 2024-03-26T02:11:20.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:19 smithi173 ceph-mon[33839]: 294.1f deep-scrub ok 2024-03-26T02:11:20.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:19 smithi072 ceph-mon[30177]: osdmap e438: 8 total, 8 up, 8 in 2024-03-26T02:11:20.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "294.1f"}]: dispatch 2024-03-26T02:11:20.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:19 smithi072 ceph-mon[30177]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "294.1f"}]: dispatch 2024-03-26T02:11:20.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:19 smithi072 ceph-mon[30177]: 294.1f deep-scrub starts 2024-03-26T02:11:20.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:19 smithi072 ceph-mon[30177]: 294.1f deep-scrub ok 2024-03-26T02:11:20.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:19 smithi072 ceph-mon[41619]: osdmap e438: 8 total, 8 up, 8 in 2024-03-26T02:11:20.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "294.1f"}]: dispatch 2024-03-26T02:11:20.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:19 smithi072 ceph-mon[41619]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "294.1f"}]: dispatch 2024-03-26T02:11:20.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:19 smithi072 ceph-mon[41619]: 294.1f deep-scrub starts 2024-03-26T02:11:20.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:19 smithi072 ceph-mon[41619]: 294.1f deep-scrub ok 2024-03-26T02:11:21.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:20 smithi173 ceph-mon[33839]: pgmap v655: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:11:21.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:11:20 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:11:21.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:20 smithi072 ceph-mon[30177]: pgmap v655: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:11:21.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:20 smithi072 ceph-mon[41619]: pgmap v655: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 667 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:11:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:22 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:11:22.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:22 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:11:22.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:22 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:11:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:23 smithi173 ceph-mon[33839]: pgmap v656: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 905 B/s wr, 114 op/s 2024-03-26T02:11:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:23 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:11:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:23 smithi072 ceph-mon[30177]: pgmap v656: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 905 B/s wr, 114 op/s 2024-03-26T02:11:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:23 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:11:23.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:23 smithi072 ceph-mon[41619]: pgmap v656: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 905 B/s wr, 114 op/s 2024-03-26T02:11:23.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:23 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:11:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:25 smithi173 ceph-mon[33839]: pgmap v657: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 769 B/s wr, 97 op/s 2024-03-26T02:11:25.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:25 smithi072 ceph-mon[41619]: pgmap v657: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 769 B/s wr, 97 op/s 2024-03-26T02:11:25.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:25 smithi072 ceph-mon[30177]: pgmap v657: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 769 B/s wr, 97 op/s 2024-03-26T02:11:25.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:11:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:11:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:11:27.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:27 smithi173 ceph-mon[33839]: pgmap v658: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 639 B/s wr, 81 op/s 2024-03-26T02:11:27.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:27 smithi072 ceph-mon[41619]: pgmap v658: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 639 B/s wr, 81 op/s 2024-03-26T02:11:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:27 smithi072 ceph-mon[30177]: pgmap v658: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 49 KiB/s rd, 639 B/s wr, 81 op/s 2024-03-26T02:11:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:29 smithi173 ceph-mon[33839]: pgmap v659: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 530 B/s wr, 67 op/s 2024-03-26T02:11:29.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:29 smithi072 ceph-mon[41619]: pgmap v659: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 530 B/s wr, 67 op/s 2024-03-26T02:11:29.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:29 smithi072 ceph-mon[30177]: pgmap v659: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 530 B/s wr, 67 op/s 2024-03-26T02:11:31.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:31 smithi173 ceph-mon[33839]: pgmap v660: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 480 B/s wr, 61 op/s 2024-03-26T02:11:31.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:11:30 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:11:31.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:31 smithi072 ceph-mon[41619]: pgmap v660: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 480 B/s wr, 61 op/s 2024-03-26T02:11:31.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:31 smithi072 ceph-mon[30177]: pgmap v660: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 480 B/s wr, 61 op/s 2024-03-26T02:11:32.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:32 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:11:32.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:32 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:11:32.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:32 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:11:33.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:33 smithi173 ceph-mon[33839]: pgmap v661: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 426 B/s wr, 54 op/s 2024-03-26T02:11:33.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:33 smithi072 ceph-mon[41619]: pgmap v661: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 426 B/s wr, 54 op/s 2024-03-26T02:11:33.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:33 smithi072 ceph-mon[30177]: pgmap v661: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 33 KiB/s rd, 426 B/s wr, 54 op/s 2024-03-26T02:11:35.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:35 smithi173 ceph-mon[33839]: pgmap v662: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:11:35.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:35 smithi072 ceph-mon[41619]: pgmap v662: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:11:35.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:35 smithi072 ceph-mon[30177]: pgmap v662: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:11:35.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:11:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:11:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:11:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:37 smithi173 ceph-mon[33839]: pgmap v663: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:11:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:37 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:11:37.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:37 smithi072 ceph-mon[30177]: pgmap v663: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:11:37.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:37 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:11:37.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:37 smithi072 ceph-mon[41619]: pgmap v663: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:11:37.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:37 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:11:39.313 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:11:39 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:11:39.014+0000 7f1d60d2e700 -1 snap_mapper.add_oid found existing snaps mapped on 294:f8778d95:test-rados-api-smithi072-82692-80::foo:2, removing 2024-03-26T02:11:39.314 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:11:39 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:11:39.014+0000 7f2ddd631700 -1 snap_mapper.add_oid found existing snaps mapped on 294:f8778d95:test-rados-api-smithi072-82692-80::foo:2, removing 2024-03-26T02:11:39.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:39 smithi173 ceph-mon[33839]: pgmap v664: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:11:39.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:39.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:39 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:39.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-79"}]: dispatch 2024-03-26T02:11:39.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:39 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-79"}]: dispatch 2024-03-26T02:11:39.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:39 smithi072 ceph-mon[41619]: pgmap v664: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:11:39.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:39.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:39 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:39.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-79"}]: dispatch 2024-03-26T02:11:39.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:39 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-79"}]: dispatch 2024-03-26T02:11:39.470 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 26 02:11:39 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-0[46061]: 2024-03-26T02:11:39.014+0000 7f18120b4700 -1 snap_mapper.add_oid found existing snaps mapped on 294:f8778d95:test-rados-api-smithi072-82692-80::foo:2, removing 2024-03-26T02:11:39.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:39 smithi072 ceph-mon[30177]: pgmap v664: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:11:39.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:39.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:39 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:39.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-79"}]: dispatch 2024-03-26T02:11:39.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:39 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-79"}]: dispatch 2024-03-26T02:11:41.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:11:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:11:41.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:41 smithi173 ceph-mon[33839]: pgmap v665: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:11:41.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:41 smithi173 ceph-mon[33839]: osdmap e439: 8 total, 8 up, 8 in 2024-03-26T02:11:41.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:41 smithi072 ceph-mon[30177]: pgmap v665: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:11:41.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:41 smithi072 ceph-mon[30177]: osdmap e439: 8 total, 8 up, 8 in 2024-03-26T02:11:41.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:41 smithi072 ceph-mon[41619]: pgmap v665: 324 pgs: 324 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:11:41.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:41 smithi072 ceph-mon[41619]: osdmap e439: 8 total, 8 up, 8 in 2024-03-26T02:11:42.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:42 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:11:42.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:42 smithi173 ceph-mon[33839]: osdmap e440: 8 total, 8 up, 8 in 2024-03-26T02:11:42.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:42 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:42.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:42 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:42 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:11:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:42 smithi072 ceph-mon[30177]: osdmap e440: 8 total, 8 up, 8 in 2024-03-26T02:11:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:42 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:42 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:42.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:42 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:11:42.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:42 smithi072 ceph-mon[41619]: osdmap e440: 8 total, 8 up, 8 in 2024-03-26T02:11:42.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:42 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:42.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:42 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:43 smithi173 ceph-mon[33839]: pgmap v668: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T02:11:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:43 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:11:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:43 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:11:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:43 smithi173 ceph-mon[33839]: osdmap e441: 8 total, 8 up, 8 in 2024-03-26T02:11:43.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:43 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:11:43.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:43 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:11:43.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:43 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:11:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:43 smithi072 ceph-mon[30177]: pgmap v668: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T02:11:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:43 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:11:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:43 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:11:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:43 smithi072 ceph-mon[30177]: osdmap e441: 8 total, 8 up, 8 in 2024-03-26T02:11:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:43 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:11:43.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:43 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:11:43.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:43 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:11:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:43 smithi072 ceph-mon[41619]: pgmap v668: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T02:11:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:43 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:11:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:43 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:11:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:43 smithi072 ceph-mon[41619]: osdmap e441: 8 total, 8 up, 8 in 2024-03-26T02:11:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:43 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:11:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:43 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:11:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:43 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:11:44.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:44 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:11:44.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:44 smithi072 ceph-mon[30177]: osdmap e442: 8 total, 8 up, 8 in 2024-03-26T02:11:44.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:44 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:44.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:44 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:44.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:44 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:11:44.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:44 smithi072 ceph-mon[41619]: osdmap e442: 8 total, 8 up, 8 in 2024-03-26T02:11:44.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:44 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:44.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:44 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:44.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:44 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:11:44.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:44 smithi173 ceph-mon[33839]: osdmap e442: 8 total, 8 up, 8 in 2024-03-26T02:11:44.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:44 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:44.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:44 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:45.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:11:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:11:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:11:45.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:45 smithi072 ceph-mon[30177]: pgmap v671: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-26T02:11:45.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:45 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:11:45.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:45 smithi072 ceph-mon[30177]: osdmap e443: 8 total, 8 up, 8 in 2024-03-26T02:11:45.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:45 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:11:45.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:45 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:11:45.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:45 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:11:45.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:45 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:11:45.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:45 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:11:45.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:45 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:11:45.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:45 smithi072 ceph-mon[41619]: pgmap v671: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-26T02:11:45.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:45 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:11:45.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:45 smithi072 ceph-mon[41619]: osdmap e443: 8 total, 8 up, 8 in 2024-03-26T02:11:45.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:45 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:11:45.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:45 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:11:45.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:45 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:11:45.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:45 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:11:45.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:45 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:11:45.472 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:45 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:11:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:45 smithi173 ceph-mon[33839]: pgmap v671: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 676 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s 2024-03-26T02:11:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:45 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:11:45.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:45 smithi173 ceph-mon[33839]: osdmap e443: 8 total, 8 up, 8 in 2024-03-26T02:11:45.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:45 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:11:45.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:45 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:11:45.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:45 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:11:45.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:45 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:11:45.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:45 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:11:45.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:45 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:11:46.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:46 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T02:11:46.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:46 smithi072 ceph-mon[30177]: osdmap e444: 8 total, 8 up, 8 in 2024-03-26T02:11:46.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:46 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:11:46.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:46 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:11:46.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:46 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T02:11:46.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:46 smithi072 ceph-mon[30177]: osdmap e445: 8 total, 8 up, 8 in 2024-03-26T02:11:46.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:46 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T02:11:46.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:46 smithi072 ceph-mon[41619]: osdmap e444: 8 total, 8 up, 8 in 2024-03-26T02:11:46.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:46 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:11:46.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:46 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:11:46.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:46 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T02:11:46.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:46 smithi072 ceph-mon[41619]: osdmap e445: 8 total, 8 up, 8 in 2024-03-26T02:11:46.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:46 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T02:11:46.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:46 smithi173 ceph-mon[33839]: osdmap e444: 8 total, 8 up, 8 in 2024-03-26T02:11:46.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:46 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:11:46.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:46 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:11:46.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:46 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T02:11:46.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:46 smithi173 ceph-mon[33839]: osdmap e445: 8 total, 8 up, 8 in 2024-03-26T02:11:47.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:47 smithi072 ceph-mon[30177]: pgmap v674: 324 pgs: 324 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:11:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:47 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T02:11:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:47 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T02:11:47.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:47 smithi072 ceph-mon[41619]: pgmap v674: 324 pgs: 324 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:11:47.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:47 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T02:11:47.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:47 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T02:11:47.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:47 smithi173 ceph-mon[33839]: pgmap v674: 324 pgs: 324 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:11:47.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:47 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T02:11:47.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:47 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T02:11:48.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:48 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-26T02:11:48.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:48 smithi072 ceph-mon[30177]: osdmap e446: 8 total, 8 up, 8 in 2024-03-26T02:11:48.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:48 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T02:11:48.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:48 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T02:11:48.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:48 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-26T02:11:48.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:48 smithi072 ceph-mon[41619]: osdmap e446: 8 total, 8 up, 8 in 2024-03-26T02:11:48.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:48 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T02:11:48.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:48 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T02:11:48.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:48 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-26T02:11:48.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:48 smithi173 ceph-mon[33839]: osdmap e446: 8 total, 8 up, 8 in 2024-03-26T02:11:48.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:48 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T02:11:48.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:48 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T02:11:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:49 smithi072 ceph-mon[30177]: pgmap v677: 324 pgs: 324 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:11:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:49 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-26T02:11:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:49 smithi072 ceph-mon[30177]: osdmap e447: 8 total, 8 up, 8 in 2024-03-26T02:11:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:49 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:11:49.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:49 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:11:49.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:49 smithi072 ceph-mon[41619]: pgmap v677: 324 pgs: 324 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:11:49.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:49 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-26T02:11:49.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:49 smithi072 ceph-mon[41619]: osdmap e447: 8 total, 8 up, 8 in 2024-03-26T02:11:49.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:49 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:11:49.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:49 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:11:49.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:49 smithi173 ceph-mon[33839]: pgmap v677: 324 pgs: 324 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:11:49.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:49 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-26T02:11:49.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:49 smithi173 ceph-mon[33839]: osdmap e447: 8 total, 8 up, 8 in 2024-03-26T02:11:49.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:49 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:11:49.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:49 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:11:50.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:50 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T02:11:50.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:50 smithi072 ceph-mon[41619]: osdmap e448: 8 total, 8 up, 8 in 2024-03-26T02:11:50.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:50 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:50.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:50 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:50.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:50 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-81"}]: dispatch 2024-03-26T02:11:50.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:50 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-81"}]: dispatch 2024-03-26T02:11:50.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:50 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T02:11:50.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:50 smithi072 ceph-mon[30177]: osdmap e448: 8 total, 8 up, 8 in 2024-03-26T02:11:50.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:50 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:50.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:50 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:50.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:50 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-81"}]: dispatch 2024-03-26T02:11:50.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:50 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-81"}]: dispatch 2024-03-26T02:11:50.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:50 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T02:11:50.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:50 smithi173 ceph-mon[33839]: osdmap e448: 8 total, 8 up, 8 in 2024-03-26T02:11:50.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:50 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:50.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:50 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:50.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:50 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-81"}]: dispatch 2024-03-26T02:11:50.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:50 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-81"}]: dispatch 2024-03-26T02:11:51.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:11:50 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:11:51.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:51 smithi173 ceph-mon[33839]: pgmap v680: 324 pgs: 324 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:11:51.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:51 smithi173 ceph-mon[33839]: osdmap e449: 8 total, 8 up, 8 in 2024-03-26T02:11:51.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:51 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:11:51.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:51 smithi072 ceph-mon[41619]: pgmap v680: 324 pgs: 324 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:11:51.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:51 smithi072 ceph-mon[41619]: osdmap e449: 8 total, 8 up, 8 in 2024-03-26T02:11:51.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:51 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:11:51.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:51 smithi072 ceph-mon[30177]: pgmap v680: 324 pgs: 324 active+clean; 8.4 MiB data, 664 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:11:51.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:51 smithi072 ceph-mon[30177]: osdmap e449: 8 total, 8 up, 8 in 2024-03-26T02:11:51.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:51 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:11:52.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:52 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:11:52.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:52 smithi072 ceph-mon[41619]: osdmap e450: 8 total, 8 up, 8 in 2024-03-26T02:11:52.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:52 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:52.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:52 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:11:52.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:52 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:11:52.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:52 smithi072 ceph-mon[30177]: osdmap e450: 8 total, 8 up, 8 in 2024-03-26T02:11:52.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:52 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:52.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:52 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:52.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:11:52.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:52 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:11:52.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:52 smithi173 ceph-mon[33839]: osdmap e450: 8 total, 8 up, 8 in 2024-03-26T02:11:52.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:52 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:52.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:52 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:11:52.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:11:53.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:53 smithi072 ceph-mon[41619]: pgmap v683: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 12 KiB/s wr, 39 op/s 2024-03-26T02:11:53.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:53 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:11:53.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:53 smithi072 ceph-mon[41619]: osdmap e451: 8 total, 8 up, 8 in 2024-03-26T02:11:53.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:53 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:11:53.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:53 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:11:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:53 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:11:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:53 smithi072 ceph-mon[30177]: pgmap v683: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 12 KiB/s wr, 39 op/s 2024-03-26T02:11:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:53 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:11:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:53 smithi072 ceph-mon[30177]: osdmap e451: 8 total, 8 up, 8 in 2024-03-26T02:11:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:53 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:11:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:53 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:11:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:53 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:11:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:53 smithi173 ceph-mon[33839]: pgmap v683: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 12 KiB/s wr, 39 op/s 2024-03-26T02:11:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:53 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:11:53.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:53 smithi173 ceph-mon[33839]: osdmap e451: 8 total, 8 up, 8 in 2024-03-26T02:11:53.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:53 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:11:53.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:53 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:11:53.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:53 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:11:54.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:54 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:11:54.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:54 smithi072 ceph-mon[41619]: osdmap e452: 8 total, 8 up, 8 in 2024-03-26T02:11:54.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:54 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:54.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:54 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:54.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:54 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:11:54.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:54 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:11:54.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:54 smithi072 ceph-mon[30177]: osdmap e452: 8 total, 8 up, 8 in 2024-03-26T02:11:54.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:54 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:54.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:54 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:54.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:54 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:11:54.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:54 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:11:54.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:54 smithi173 ceph-mon[33839]: osdmap e452: 8 total, 8 up, 8 in 2024-03-26T02:11:54.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:54 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:54.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:54 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:54.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:54 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:11:55.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:11:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:11:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:11:55.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:55 smithi072 ceph-mon[41619]: pgmap v686: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 12 KiB/s wr, 39 op/s 2024-03-26T02:11:55.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:55 smithi072 ceph-mon[41619]: osdmap e453: 8 total, 8 up, 8 in 2024-03-26T02:11:55.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:55 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:11:55.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:55 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:11:55.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:55 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T02:11:55.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:55 smithi072 ceph-mon[41619]: osdmap e454: 8 total, 8 up, 8 in 2024-03-26T02:11:55.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:55 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:11:55.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:55 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:11:55.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:55 smithi072 ceph-mon[30177]: pgmap v686: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 12 KiB/s wr, 39 op/s 2024-03-26T02:11:55.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:55 smithi072 ceph-mon[30177]: osdmap e453: 8 total, 8 up, 8 in 2024-03-26T02:11:55.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:55 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:11:55.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:55 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:11:55.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:55 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T02:11:55.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:55 smithi072 ceph-mon[30177]: osdmap e454: 8 total, 8 up, 8 in 2024-03-26T02:11:55.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:55 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:11:55.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:55 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:11:55.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:55 smithi173 ceph-mon[33839]: pgmap v686: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 12 KiB/s wr, 39 op/s 2024-03-26T02:11:55.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:55 smithi173 ceph-mon[33839]: osdmap e453: 8 total, 8 up, 8 in 2024-03-26T02:11:55.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:55 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:11:55.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:55 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:11:55.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:55 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T02:11:55.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:55 smithi173 ceph-mon[33839]: osdmap e454: 8 total, 8 up, 8 in 2024-03-26T02:11:55.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:55 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:11:55.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:55 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:11:57.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:57 smithi072 ceph-mon[41619]: pgmap v689: 324 pgs: 324 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:11:57.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:57 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T02:11:57.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:57 smithi072 ceph-mon[41619]: osdmap e455: 8 total, 8 up, 8 in 2024-03-26T02:11:57.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:57 smithi072 ceph-mon[30177]: pgmap v689: 324 pgs: 324 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:11:57.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:57 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T02:11:57.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:57 smithi072 ceph-mon[30177]: osdmap e455: 8 total, 8 up, 8 in 2024-03-26T02:11:57.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:57 smithi173 ceph-mon[33839]: pgmap v689: 324 pgs: 324 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:11:57.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:57 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T02:11:57.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:57 smithi173 ceph-mon[33839]: osdmap e455: 8 total, 8 up, 8 in 2024-03-26T02:11:58.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:58 smithi072 ceph-mon[30177]: osdmap e456: 8 total, 8 up, 8 in 2024-03-26T02:11:58.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:58 smithi072 ceph-mon[41619]: osdmap e456: 8 total, 8 up, 8 in 2024-03-26T02:11:58.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:58 smithi173 ceph-mon[33839]: osdmap e456: 8 total, 8 up, 8 in 2024-03-26T02:11:59.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:59 smithi072 ceph-mon[30177]: pgmap v692: 324 pgs: 324 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:11:59.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:59 smithi072 ceph-mon[30177]: osdmap e457: 8 total, 8 up, 8 in 2024-03-26T02:11:59.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:59 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:59.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:59 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:59.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:59 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-83"}]: dispatch 2024-03-26T02:11:59.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:11:59 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-83"}]: dispatch 2024-03-26T02:11:59.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:59 smithi072 ceph-mon[41619]: pgmap v692: 324 pgs: 324 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:11:59.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:59 smithi072 ceph-mon[41619]: osdmap e457: 8 total, 8 up, 8 in 2024-03-26T02:11:59.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:59 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:59.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:59 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:59.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:59 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-83"}]: dispatch 2024-03-26T02:11:59.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:11:59 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-83"}]: dispatch 2024-03-26T02:11:59.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:59 smithi173 ceph-mon[33839]: pgmap v692: 324 pgs: 324 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:11:59.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:59 smithi173 ceph-mon[33839]: osdmap e457: 8 total, 8 up, 8 in 2024-03-26T02:11:59.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:59 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:59.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:59 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:11:59.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:59 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-83"}]: dispatch 2024-03-26T02:11:59.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:11:59 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-83"}]: dispatch 2024-03-26T02:12:00.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:00 smithi072 ceph-mon[30177]: osdmap e458: 8 total, 8 up, 8 in 2024-03-26T02:12:00.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:00 smithi072 ceph-mon[41619]: osdmap e458: 8 total, 8 up, 8 in 2024-03-26T02:12:00.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:00 smithi173 ceph-mon[33839]: osdmap e458: 8 total, 8 up, 8 in 2024-03-26T02:12:01.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:12:00 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:12:01.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:01 smithi173 ceph-mon[33839]: pgmap v695: 292 pgs: 292 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:12:01.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:01 smithi173 ceph-mon[33839]: osdmap e459: 8 total, 8 up, 8 in 2024-03-26T02:12:01.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:01.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:01 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:01.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:01 smithi072 ceph-mon[30177]: pgmap v695: 292 pgs: 292 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:12:01.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:01 smithi072 ceph-mon[30177]: osdmap e459: 8 total, 8 up, 8 in 2024-03-26T02:12:01.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:01.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:01 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:01.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:01 smithi072 ceph-mon[41619]: pgmap v695: 292 pgs: 292 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:12:01.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:01 smithi072 ceph-mon[41619]: osdmap e459: 8 total, 8 up, 8 in 2024-03-26T02:12:01.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:01.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:01 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:02.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:02 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:12:02.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:02 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:12:02.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:02 smithi072 ceph-mon[30177]: osdmap e460: 8 total, 8 up, 8 in 2024-03-26T02:12:02.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:12:02.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:12:02.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:02 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:12:02.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:02 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:12:02.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:02 smithi072 ceph-mon[30177]: osdmap e461: 8 total, 8 up, 8 in 2024-03-26T02:12:02.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:02.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:02 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:02.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:02 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:12:02.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:02 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:12:02.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:02 smithi072 ceph-mon[41619]: osdmap e460: 8 total, 8 up, 8 in 2024-03-26T02:12:02.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:12:02.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:12:02.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:02 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:12:02.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:02 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:12:02.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:02 smithi072 ceph-mon[41619]: osdmap e461: 8 total, 8 up, 8 in 2024-03-26T02:12:02.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:02.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:02 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:02.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:02 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:12:02.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:02 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:12:02.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:02 smithi173 ceph-mon[33839]: osdmap e460: 8 total, 8 up, 8 in 2024-03-26T02:12:02.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:12:02.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:12:02.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:02 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:12:02.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:02 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:12:02.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:02 smithi173 ceph-mon[33839]: osdmap e461: 8 total, 8 up, 8 in 2024-03-26T02:12:02.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:02.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:02 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:03 smithi072 ceph-mon[30177]: pgmap v698: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-26T02:12:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:03 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:12:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:03 smithi072 ceph-mon[30177]: osdmap e462: 8 total, 8 up, 8 in 2024-03-26T02:12:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:12:03.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:03 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:12:03.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:03 smithi072 ceph-mon[41619]: pgmap v698: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-26T02:12:03.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:03 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:12:03.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:03 smithi072 ceph-mon[41619]: osdmap e462: 8 total, 8 up, 8 in 2024-03-26T02:12:03.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:12:03.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:03 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:12:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:03 smithi173 ceph-mon[33839]: pgmap v698: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-26T02:12:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:03 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_tier","val": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:12:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:03 smithi173 ceph-mon[33839]: osdmap e462: 8 total, 8 up, 8 in 2024-03-26T02:12:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:12:03.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:03 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:12:05.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:12:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:12:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:12:05.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:05 smithi072 ceph-mon[30177]: pgmap v701: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-26T02:12:05.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T02:12:05.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:05 smithi072 ceph-mon[30177]: osdmap e463: 8 total, 8 up, 8 in 2024-03-26T02:12:05.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:12:05.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:05 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:12:05.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:05 smithi072 ceph-mon[41619]: pgmap v701: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-26T02:12:05.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T02:12:05.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:05 smithi072 ceph-mon[41619]: osdmap e463: 8 total, 8 up, 8 in 2024-03-26T02:12:05.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:12:05.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:05 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:12:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:05 smithi173 ceph-mon[33839]: pgmap v701: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-26T02:12:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T02:12:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:05 smithi173 ceph-mon[33839]: osdmap e463: 8 total, 8 up, 8 in 2024-03-26T02:12:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:12:05.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:05 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:12:06.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:06 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T02:12:06.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:06 smithi072 ceph-mon[30177]: osdmap e464: 8 total, 8 up, 8 in 2024-03-26T02:12:06.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:06 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T02:12:06.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:06 smithi072 ceph-mon[41619]: osdmap e464: 8 total, 8 up, 8 in 2024-03-26T02:12:06.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:06 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T02:12:06.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:06 smithi173 ceph-mon[33839]: osdmap e464: 8 total, 8 up, 8 in 2024-03-26T02:12:07.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:07 smithi072 ceph-mon[30177]: pgmap v704: 324 pgs: 324 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-26T02:12:07.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:07 smithi072 ceph-mon[30177]: osdmap e465: 8 total, 8 up, 8 in 2024-03-26T02:12:07.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:07 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:12:07.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:07 smithi072 ceph-mon[41619]: pgmap v704: 324 pgs: 324 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-26T02:12:07.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:07 smithi072 ceph-mon[41619]: osdmap e465: 8 total, 8 up, 8 in 2024-03-26T02:12:07.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:07 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:12:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:07 smithi173 ceph-mon[33839]: pgmap v704: 324 pgs: 324 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-26T02:12:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:07 smithi173 ceph-mon[33839]: osdmap e465: 8 total, 8 up, 8 in 2024-03-26T02:12:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:07 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:12:08.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:08 smithi072 ceph-mon[30177]: osdmap e466: 8 total, 8 up, 8 in 2024-03-26T02:12:08.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:08.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:08.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-85"}]: dispatch 2024-03-26T02:12:08.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:08 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-85"}]: dispatch 2024-03-26T02:12:08.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:08 smithi072 ceph-mon[41619]: osdmap e466: 8 total, 8 up, 8 in 2024-03-26T02:12:08.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:08.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:08.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-85"}]: dispatch 2024-03-26T02:12:08.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:08 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-85"}]: dispatch 2024-03-26T02:12:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:08 smithi173 ceph-mon[33839]: osdmap e466: 8 total, 8 up, 8 in 2024-03-26T02:12:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-85"}]: dispatch 2024-03-26T02:12:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:08 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-85"}]: dispatch 2024-03-26T02:12:09.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:09 smithi072 ceph-mon[30177]: pgmap v707: 324 pgs: 324 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-26T02:12:09.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:09 smithi072 ceph-mon[30177]: osdmap e467: 8 total, 8 up, 8 in 2024-03-26T02:12:09.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:09 smithi072 ceph-mon[41619]: pgmap v707: 324 pgs: 324 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-26T02:12:09.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:09 smithi072 ceph-mon[41619]: osdmap e467: 8 total, 8 up, 8 in 2024-03-26T02:12:09.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:09 smithi173 ceph-mon[33839]: pgmap v707: 324 pgs: 324 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-26T02:12:09.566 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:09 smithi173 ceph-mon[33839]: osdmap e467: 8 total, 8 up, 8 in 2024-03-26T02:12:10.223 INFO:tasks.workunit.client.0.smithi072.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (18079 ms) 2024-03-26T02:12:10.223 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-03-26T02:12:10.223 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (22096 ms) 2024-03-26T02:12:10.223 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-03-26T02:12:10.223 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3008 ms) 2024-03-26T02:12:10.223 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-03-26T02:12:10.223 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T02:12:10.223 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3034 ms) 2024-03-26T02:12:10.223 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-03-26T02:12:10.223 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3015 ms) 2024-03-26T02:12:10.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-03-26T02:12:10.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T02:12:10.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3031 ms) 2024-03-26T02:12:10.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-03-26T02:12:10.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T02:12:10.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4025 ms) 2024-03-26T02:12:10.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-03-26T02:12:10.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T02:12:10.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (39529 ms) 2024-03-26T02:12:10.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-03-26T02:12:10.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T02:12:10.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (16959 ms) 2024-03-26T02:12:10.224 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-03-26T02:12:10.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T02:12:10.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4034 ms) 2024-03-26T02:12:10.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-03-26T02:12:10.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T02:12:10.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3009 ms) 2024-03-26T02:12:10.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-03-26T02:12:10.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T02:12:10.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (25090 ms) 2024-03-26T02:12:10.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-03-26T02:12:10.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T02:12:10.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15086 ms) 2024-03-26T02:12:10.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-03-26T02:12:10.225 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T02:12:10.226 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5041 ms) 2024-03-26T02:12:10.226 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-03-26T02:12:10.226 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T02:12:10.226 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (4028 ms) 2024-03-26T02:12:10.226 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-03-26T02:12:10.226 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T02:12:10.226 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: waiting for scrubs... 2024-03-26T02:12:10.226 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: done waiting 2024-03-26T02:12:10.226 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25068 ms) 2024-03-26T02:12:10.226 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-03-26T02:12:10.226 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T02:12:10.226 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (10065 ms) 2024-03-26T02:12:10.227 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-03-26T02:12:10.227 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T02:12:10.227 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9050 ms) 2024-03-26T02:12:10.227 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-03-26T02:12:10.227 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T02:12:10.227 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9043 ms) 2024-03-26T02:12:10.227 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-03-26T02:12:10.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:10 smithi072 ceph-mon[41619]: osdmap e468: 8 total, 8 up, 8 in 2024-03-26T02:12:10.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:10.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:10.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:10 smithi072 ceph-mon[30177]: osdmap e468: 8 total, 8 up, 8 in 2024-03-26T02:12:10.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:10.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:10.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:10 smithi173 ceph-mon[33839]: osdmap e468: 8 total, 8 up, 8 in 2024-03-26T02:12:10.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:10.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:11.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:12:10 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:12:11.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:11 smithi173 ceph-mon[33839]: pgmap v710: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:12:11.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:11 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:12:11.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:11 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:12:11.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:11 smithi173 ceph-mon[33839]: osdmap e469: 8 total, 8 up, 8 in 2024-03-26T02:12:11.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:11 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:12:11.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:11 smithi072 ceph-mon[30177]: pgmap v710: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:12:11.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:11 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:12:11.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:11 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:12:11.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:11 smithi072 ceph-mon[30177]: osdmap e469: 8 total, 8 up, 8 in 2024-03-26T02:12:11.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:11 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:12:11.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:11 smithi072 ceph-mon[41619]: pgmap v710: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:12:11.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:11 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:12:11.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:11 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:12:11.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:11 smithi072 ceph-mon[41619]: osdmap e469: 8 total, 8 up, 8 in 2024-03-26T02:12:11.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:11 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:12:12.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:12 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:12:12.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:12 smithi173 ceph-mon[33839]: osdmap e470: 8 total, 8 up, 8 in 2024-03-26T02:12:12.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:12:12.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:12:12.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:12 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:12:12.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:12 smithi072 ceph-mon[30177]: osdmap e470: 8 total, 8 up, 8 in 2024-03-26T02:12:12.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:12:12.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:12:12.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:12 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:12:12.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:12 smithi072 ceph-mon[41619]: osdmap e470: 8 total, 8 up, 8 in 2024-03-26T02:12:12.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:12:12.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:12:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:13 smithi173 ceph-mon[33839]: pgmap v713: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-03-26T02:12:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:13 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:12:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:13 smithi173 ceph-mon[33839]: osdmap e471: 8 total, 8 up, 8 in 2024-03-26T02:12:13.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:13 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_tier","val": "test-rados-api-smithi072-82692-89-test-flush"}]: dispatch 2024-03-26T02:12:13.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:13 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_tier","val": "test-rados-api-smithi072-82692-89-test-flush"}]: dispatch 2024-03-26T02:12:13.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:13 smithi072 ceph-mon[30177]: pgmap v713: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-03-26T02:12:13.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:13 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:12:13.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:13 smithi072 ceph-mon[30177]: osdmap e471: 8 total, 8 up, 8 in 2024-03-26T02:12:13.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:13 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_tier","val": "test-rados-api-smithi072-82692-89-test-flush"}]: dispatch 2024-03-26T02:12:13.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:13 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_tier","val": "test-rados-api-smithi072-82692-89-test-flush"}]: dispatch 2024-03-26T02:12:13.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:13 smithi072 ceph-mon[41619]: pgmap v713: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 2.5 KiB/s wr, 10 op/s 2024-03-26T02:12:13.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:13 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:12:13.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:13 smithi072 ceph-mon[41619]: osdmap e471: 8 total, 8 up, 8 in 2024-03-26T02:12:13.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:13 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_tier","val": "test-rados-api-smithi072-82692-89-test-flush"}]: dispatch 2024-03-26T02:12:13.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:13 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_tier","val": "test-rados-api-smithi072-82692-89-test-flush"}]: dispatch 2024-03-26T02:12:14.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:14 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_tier","val": "test-rados-api-smithi072-82692-89-test-flush"}]': finished 2024-03-26T02:12:14.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:14 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:12:14.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:14 smithi173 ceph-mon[33839]: osdmap e472: 8 total, 8 up, 8 in 2024-03-26T02:12:14.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:14 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:12:14.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:14 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T02:12:14.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:14 smithi173 ceph-mon[33839]: osdmap e473: 8 total, 8 up, 8 in 2024-03-26T02:12:14.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:14 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:12:14.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:14 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:12:14.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:14 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_tier","val": "test-rados-api-smithi072-82692-89-test-flush"}]': finished 2024-03-26T02:12:14.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:14 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:12:14.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:14 smithi072 ceph-mon[30177]: osdmap e472: 8 total, 8 up, 8 in 2024-03-26T02:12:14.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:14 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:12:14.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:14 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T02:12:14.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:14 smithi072 ceph-mon[30177]: osdmap e473: 8 total, 8 up, 8 in 2024-03-26T02:12:14.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:14 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:12:14.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:14 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:12:14.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:14 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_tier","val": "test-rados-api-smithi072-82692-89-test-flush"}]': finished 2024-03-26T02:12:14.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:14 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:12:14.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:14 smithi072 ceph-mon[41619]: osdmap e472: 8 total, 8 up, 8 in 2024-03-26T02:12:14.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:14 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:12:14.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:14 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T02:12:14.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:14 smithi072 ceph-mon[41619]: osdmap e473: 8 total, 8 up, 8 in 2024-03-26T02:12:14.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:14 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:12:14.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:14 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:12:15.245 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:12:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:12:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:12:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:15 smithi173 ceph-mon[33839]: pgmap v716: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-03-26T02:12:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T02:12:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:15 smithi173 ceph-mon[33839]: osdmap e474: 8 total, 8 up, 8 in 2024-03-26T02:12:15.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:15 smithi072 ceph-mon[30177]: pgmap v716: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-03-26T02:12:15.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T02:12:15.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:15 smithi072 ceph-mon[30177]: osdmap e474: 8 total, 8 up, 8 in 2024-03-26T02:12:15.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:15 smithi072 ceph-mon[41619]: pgmap v716: 356 pgs: 32 unknown, 324 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.5 KiB/s wr, 10 op/s 2024-03-26T02:12:15.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T02:12:15.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:15 smithi072 ceph-mon[41619]: osdmap e474: 8 total, 8 up, 8 in 2024-03-26T02:12:17.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:17 smithi173 ceph-mon[33839]: pgmap v719: 356 pgs: 356 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-26T02:12:17.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:17 smithi173 ceph-mon[33839]: osdmap e475: 8 total, 8 up, 8 in 2024-03-26T02:12:17.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:17 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:17.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:17 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:17.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:17 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-87"}]: dispatch 2024-03-26T02:12:17.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:17 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-87"}]: dispatch 2024-03-26T02:12:17.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:17 smithi072 ceph-mon[30177]: pgmap v719: 356 pgs: 356 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-26T02:12:17.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:17 smithi072 ceph-mon[30177]: osdmap e475: 8 total, 8 up, 8 in 2024-03-26T02:12:17.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:17 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:17.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:17 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:17.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:17 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-87"}]: dispatch 2024-03-26T02:12:17.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:17 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-87"}]: dispatch 2024-03-26T02:12:17.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:17 smithi072 ceph-mon[41619]: pgmap v719: 356 pgs: 356 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 2 op/s 2024-03-26T02:12:17.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:17 smithi072 ceph-mon[41619]: osdmap e475: 8 total, 8 up, 8 in 2024-03-26T02:12:17.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:17 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:17.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:17 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:17.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:17 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-87"}]: dispatch 2024-03-26T02:12:17.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:17 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-87"}]: dispatch 2024-03-26T02:12:18.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:18 smithi173 ceph-mon[33839]: osdmap e476: 8 total, 8 up, 8 in 2024-03-26T02:12:18.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:18 smithi072 ceph-mon[30177]: osdmap e476: 8 total, 8 up, 8 in 2024-03-26T02:12:18.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:18 smithi072 ceph-mon[41619]: osdmap e476: 8 total, 8 up, 8 in 2024-03-26T02:12:19.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:19 smithi173 ceph-mon[33839]: pgmap v722: 292 pgs: 292 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:12:19.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:19 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:12:19.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:19 smithi173 ceph-mon[33839]: osdmap e477: 8 total, 8 up, 8 in 2024-03-26T02:12:19.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:19.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:19 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:19.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:19 smithi072 ceph-mon[30177]: pgmap v722: 292 pgs: 292 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:12:19.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:19 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:12:19.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:19 smithi072 ceph-mon[30177]: osdmap e477: 8 total, 8 up, 8 in 2024-03-26T02:12:19.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:19.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:19 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:19.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:19 smithi072 ceph-mon[41619]: pgmap v722: 292 pgs: 292 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:12:19.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:19 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:12:19.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:19 smithi072 ceph-mon[41619]: osdmap e477: 8 total, 8 up, 8 in 2024-03-26T02:12:19.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:19.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:19 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:20.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:20 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:12:20.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:20 smithi173 ceph-mon[33839]: osdmap e478: 8 total, 8 up, 8 in 2024-03-26T02:12:20.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:12:20.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:12:20.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:20 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:12:20.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:20 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:12:20.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:20 smithi072 ceph-mon[30177]: osdmap e478: 8 total, 8 up, 8 in 2024-03-26T02:12:20.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:12:20.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:12:20.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:20 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:12:20.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:20 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:12:20.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:20 smithi072 ceph-mon[41619]: osdmap e478: 8 total, 8 up, 8 in 2024-03-26T02:12:20.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:12:20.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:12:20.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:20 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:12:21.283 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:12:21 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:12:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:21 smithi173 ceph-mon[33839]: pgmap v725: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:12:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:21 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:12:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:21 smithi173 ceph-mon[33839]: osdmap e479: 8 total, 8 up, 8 in 2024-03-26T02:12:21.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:21 smithi072 ceph-mon[41619]: pgmap v725: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:12:21.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:21 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:12:21.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:21 smithi072 ceph-mon[41619]: osdmap e479: 8 total, 8 up, 8 in 2024-03-26T02:12:21.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:21 smithi072 ceph-mon[30177]: pgmap v725: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:12:21.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:21 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:12:21.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:21 smithi072 ceph-mon[30177]: osdmap e479: 8 total, 8 up, 8 in 2024-03-26T02:12:22.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:22 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:12:22.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:22 smithi173 ceph-mon[33839]: osdmap e480: 8 total, 8 up, 8 in 2024-03-26T02:12:22.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:22 smithi173 ceph-mon[33839]: pgmap v728: 324 pgs: 324 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-26T02:12:22.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:22 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:12:22.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:22 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:12:22.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:22 smithi072 ceph-mon[41619]: osdmap e480: 8 total, 8 up, 8 in 2024-03-26T02:12:22.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:22 smithi072 ceph-mon[41619]: pgmap v728: 324 pgs: 324 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-26T02:12:22.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:22 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:12:22.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:22 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:12:22.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:22 smithi072 ceph-mon[30177]: osdmap e480: 8 total, 8 up, 8 in 2024-03-26T02:12:22.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:22 smithi072 ceph-mon[30177]: pgmap v728: 324 pgs: 324 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.2 KiB/s wr, 4 op/s 2024-03-26T02:12:22.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:22 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:12:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:23 smithi173 ceph-mon[33839]: osdmap e481: 8 total, 8 up, 8 in 2024-03-26T02:12:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:23 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:23 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:23.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:23 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-90"}]: dispatch 2024-03-26T02:12:23.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:23 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-90"}]: dispatch 2024-03-26T02:12:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:23 smithi072 ceph-mon[30177]: osdmap e481: 8 total, 8 up, 8 in 2024-03-26T02:12:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:23 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:23 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:23 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-90"}]: dispatch 2024-03-26T02:12:23.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:23 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-90"}]: dispatch 2024-03-26T02:12:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:23 smithi072 ceph-mon[41619]: osdmap e481: 8 total, 8 up, 8 in 2024-03-26T02:12:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:23 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:23 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:23 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-90"}]: dispatch 2024-03-26T02:12:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:23 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-90"}]: dispatch 2024-03-26T02:12:24.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:24 smithi173 ceph-mon[33839]: osdmap e482: 8 total, 8 up, 8 in 2024-03-26T02:12:24.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:24 smithi173 ceph-mon[33839]: pgmap v731: 292 pgs: 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:12:24.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:24 smithi072 ceph-mon[30177]: osdmap e482: 8 total, 8 up, 8 in 2024-03-26T02:12:24.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:24 smithi072 ceph-mon[30177]: pgmap v731: 292 pgs: 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:12:24.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:24 smithi072 ceph-mon[41619]: osdmap e482: 8 total, 8 up, 8 in 2024-03-26T02:12:24.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:24 smithi072 ceph-mon[41619]: pgmap v731: 292 pgs: 292 active+clean; 8.4 MiB data, 641 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:12:25.329 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:12:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:12:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:12:25.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:25 smithi072 ceph-mon[30177]: osdmap e483: 8 total, 8 up, 8 in 2024-03-26T02:12:25.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:25.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:25 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:25.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:25 smithi072 ceph-mon[41619]: osdmap e483: 8 total, 8 up, 8 in 2024-03-26T02:12:25.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:25.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:25 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:25.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:25 smithi173 ceph-mon[33839]: osdmap e483: 8 total, 8 up, 8 in 2024-03-26T02:12:25.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:25.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:25 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:26.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:26 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:12:26.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:26 smithi072 ceph-mon[30177]: osdmap e484: 8 total, 8 up, 8 in 2024-03-26T02:12:26.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:26 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:12:26.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:26 smithi072 ceph-mon[30177]: pgmap v734: 324 pgs: 324 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:12:26.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:26 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:12:26.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:26 smithi072 ceph-mon[41619]: osdmap e484: 8 total, 8 up, 8 in 2024-03-26T02:12:26.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:26 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:12:26.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:26 smithi072 ceph-mon[41619]: pgmap v734: 324 pgs: 324 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:12:26.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:26 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:12:26.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:26 smithi173 ceph-mon[33839]: osdmap e484: 8 total, 8 up, 8 in 2024-03-26T02:12:26.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:26 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:12:26.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:26 smithi173 ceph-mon[33839]: pgmap v734: 324 pgs: 324 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:12:27.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:27 smithi072 ceph-mon[30177]: osdmap e485: 8 total, 8 up, 8 in 2024-03-26T02:12:27.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:27 smithi072 ceph-mon[41619]: osdmap e485: 8 total, 8 up, 8 in 2024-03-26T02:12:27.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:27 smithi173 ceph-mon[33839]: osdmap e485: 8 total, 8 up, 8 in 2024-03-26T02:12:28.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:28 smithi072 ceph-mon[30177]: osdmap e486: 8 total, 8 up, 8 in 2024-03-26T02:12:28.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:28 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:28.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:28 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:28.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:28 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-92"}]: dispatch 2024-03-26T02:12:28.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:28 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-92"}]: dispatch 2024-03-26T02:12:28.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:28 smithi072 ceph-mon[30177]: pgmap v737: 324 pgs: 324 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-03-26T02:12:28.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:28 smithi072 ceph-mon[41619]: osdmap e486: 8 total, 8 up, 8 in 2024-03-26T02:12:28.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:28 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:28.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:28 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:28.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:28 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-92"}]: dispatch 2024-03-26T02:12:28.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:28 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-92"}]: dispatch 2024-03-26T02:12:28.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:28 smithi072 ceph-mon[41619]: pgmap v737: 324 pgs: 324 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-03-26T02:12:28.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:28 smithi173 ceph-mon[33839]: osdmap e486: 8 total, 8 up, 8 in 2024-03-26T02:12:28.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:28 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:28.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:28 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:28.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:28 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-92"}]: dispatch 2024-03-26T02:12:28.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:28 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-92"}]: dispatch 2024-03-26T02:12:28.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:28 smithi173 ceph-mon[33839]: pgmap v737: 324 pgs: 324 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-03-26T02:12:29.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:29 smithi072 ceph-mon[30177]: osdmap e487: 8 total, 8 up, 8 in 2024-03-26T02:12:29.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:29 smithi072 ceph-mon[41619]: osdmap e487: 8 total, 8 up, 8 in 2024-03-26T02:12:29.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:29 smithi173 ceph-mon[33839]: osdmap e487: 8 total, 8 up, 8 in 2024-03-26T02:12:30.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:30 smithi072 ceph-mon[30177]: osdmap e488: 8 total, 8 up, 8 in 2024-03-26T02:12:30.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:30.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:30.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:30 smithi072 ceph-mon[30177]: pgmap v740: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:12:30.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:30 smithi072 ceph-mon[41619]: osdmap e488: 8 total, 8 up, 8 in 2024-03-26T02:12:30.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:30.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:30.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:30 smithi072 ceph-mon[41619]: pgmap v740: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:12:30.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:30 smithi173 ceph-mon[33839]: osdmap e488: 8 total, 8 up, 8 in 2024-03-26T02:12:30.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:30.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:30.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:30 smithi173 ceph-mon[33839]: pgmap v740: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:12:31.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:12:31 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:12:31.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:31 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:12:31.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:31 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:12:31.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:31 smithi072 ceph-mon[30177]: osdmap e489: 8 total, 8 up, 8 in 2024-03-26T02:12:31.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:31 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:12:31.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:31 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:12:31.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:31 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:12:31.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:31 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:12:31.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:31 smithi072 ceph-mon[41619]: osdmap e489: 8 total, 8 up, 8 in 2024-03-26T02:12:31.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:31 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:12:31.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:31 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:12:31.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:31 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:12:31.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:31 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:12:31.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:31 smithi173 ceph-mon[33839]: osdmap e489: 8 total, 8 up, 8 in 2024-03-26T02:12:31.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:31 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:12:31.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:31 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:12:32.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:32 smithi072 ceph-mon[30177]: osdmap e490: 8 total, 8 up, 8 in 2024-03-26T02:12:32.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:32 smithi072 ceph-mon[30177]: pgmap v743: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T02:12:32.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:32 smithi072 ceph-mon[41619]: osdmap e490: 8 total, 8 up, 8 in 2024-03-26T02:12:32.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:32 smithi072 ceph-mon[41619]: pgmap v743: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T02:12:32.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:32 smithi173 ceph-mon[33839]: osdmap e490: 8 total, 8 up, 8 in 2024-03-26T02:12:32.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:32 smithi173 ceph-mon[33839]: pgmap v743: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T02:12:33.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:33 smithi072 ceph-mon[30177]: osdmap e491: 8 total, 8 up, 8 in 2024-03-26T02:12:33.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:33 smithi072 ceph-mon[41619]: osdmap e491: 8 total, 8 up, 8 in 2024-03-26T02:12:33.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:33 smithi173 ceph-mon[33839]: osdmap e491: 8 total, 8 up, 8 in 2024-03-26T02:12:34.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:34 smithi072 ceph-mon[30177]: pgmap v745: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.9 KiB/s wr, 3 op/s 2024-03-26T02:12:34.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:34 smithi072 ceph-mon[41619]: pgmap v745: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.9 KiB/s wr, 3 op/s 2024-03-26T02:12:34.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:34 smithi173 ceph-mon[33839]: pgmap v745: 324 pgs: 324 active+clean; 8.4 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.9 KiB/s wr, 3 op/s 2024-03-26T02:12:35.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:12:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:12:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:12:36.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:35 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:12:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:35 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:12:36.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:35 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:12:37.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:36 smithi173 ceph-mon[33839]: pgmap v746: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-26T02:12:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:36 smithi072 ceph-mon[30177]: pgmap v746: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-26T02:12:37.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:36 smithi072 ceph-mon[41619]: pgmap v746: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-26T02:12:38.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:37 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:12:38.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:37 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:12:38.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:37 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:12:39.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:38 smithi173 ceph-mon[33839]: pgmap v747: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-26T02:12:39.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:38 smithi072 ceph-mon[30177]: pgmap v747: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-26T02:12:39.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:38 smithi072 ceph-mon[41619]: pgmap v747: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-26T02:12:41.021 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:40 smithi072 ceph-mon[30177]: pgmap v748: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 246 B/s wr, 3 op/s 2024-03-26T02:12:41.022 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:40 smithi072 ceph-mon[41619]: pgmap v748: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 246 B/s wr, 3 op/s 2024-03-26T02:12:41.022 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:40 smithi173 ceph-mon[33839]: pgmap v748: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 246 B/s wr, 3 op/s 2024-03-26T02:12:41.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:12:41 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:12:42.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:41 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:12:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:41 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:12:42.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:41 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:12:43.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:42 smithi173 ceph-mon[33839]: pgmap v749: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-26T02:12:43.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:42 smithi072 ceph-mon[30177]: pgmap v749: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-26T02:12:43.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:42 smithi072 ceph-mon[41619]: pgmap v749: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-26T02:12:44.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:43 smithi173 ceph-mon[33839]: osdmap e492: 8 total, 8 up, 8 in 2024-03-26T02:12:44.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:43 smithi072 ceph-mon[41619]: osdmap e492: 8 total, 8 up, 8 in 2024-03-26T02:12:44.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:43 smithi072 ceph-mon[30177]: osdmap e492: 8 total, 8 up, 8 in 2024-03-26T02:12:45.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:44 smithi173 ceph-mon[33839]: pgmap v751: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-26T02:12:45.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:44 smithi072 ceph-mon[30177]: pgmap v751: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-26T02:12:45.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:44 smithi072 ceph-mon[41619]: pgmap v751: 324 pgs: 324 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-26T02:12:45.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:12:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:12:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:12:46.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:45 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:12:46.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:45 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:12:46.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:45 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:12:46.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:45 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:12:46.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:45 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:12:46.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:45 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:12:46.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:45 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:12:46.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:45 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:12:46.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:45 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:12:46.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:45 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:12:46.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:45 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:12:46.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:45 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:12:46.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:45 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:12:46.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:45 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:12:46.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:12:46.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:12:46.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:12:46.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:12:46.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:12:46.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:12:46.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:12:46.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:12:46.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:12:46.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:12:46.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:12:46.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:12:46.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:12:46.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:12:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:12:46.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:12:46.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:12:46.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:12:46.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:12:46.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:12:46.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:12:46.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:12:46.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T02:12:46.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:12:46.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:12:46.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:12:46.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:12:46.222 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:45 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:12:47.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:46 smithi173 ceph-mon[33839]: Adjusting osd_memory_target on smithi072 to 3248M 2024-03-26T02:12:47.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:46 smithi173 ceph-mon[33839]: Adjusting osd_memory_target on smithi173 to 3248M 2024-03-26T02:12:47.066 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:46 smithi173 ceph-mon[33839]: pgmap v752: 324 pgs: 324 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:12:47.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:46 smithi072 ceph-mon[30177]: Adjusting osd_memory_target on smithi072 to 3248M 2024-03-26T02:12:47.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:46 smithi072 ceph-mon[30177]: Adjusting osd_memory_target on smithi173 to 3248M 2024-03-26T02:12:47.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:46 smithi072 ceph-mon[30177]: pgmap v752: 324 pgs: 324 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:12:47.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:46 smithi072 ceph-mon[41619]: Adjusting osd_memory_target on smithi072 to 3248M 2024-03-26T02:12:47.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:46 smithi072 ceph-mon[41619]: Adjusting osd_memory_target on smithi173 to 3248M 2024-03-26T02:12:47.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:46 smithi072 ceph-mon[41619]: pgmap v752: 324 pgs: 324 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:12:49.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:48 smithi173 ceph-mon[33839]: pgmap v753: 324 pgs: 324 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:12:49.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:48 smithi072 ceph-mon[30177]: pgmap v753: 324 pgs: 324 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:12:49.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:48 smithi072 ceph-mon[41619]: pgmap v753: 324 pgs: 324 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:12:51.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:50 smithi173 ceph-mon[33839]: pgmap v754: 324 pgs: 324 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:12:51.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:50 smithi173 ceph-mon[33839]: osdmap e493: 8 total, 8 up, 8 in 2024-03-26T02:12:51.064 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:12:51 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:12:51.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:50 smithi072 ceph-mon[30177]: pgmap v754: 324 pgs: 324 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:12:51.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:50 smithi072 ceph-mon[30177]: osdmap e493: 8 total, 8 up, 8 in 2024-03-26T02:12:51.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:50 smithi072 ceph-mon[41619]: pgmap v754: 324 pgs: 324 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:12:51.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:50 smithi072 ceph-mon[41619]: osdmap e493: 8 total, 8 up, 8 in 2024-03-26T02:12:52.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:51 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:12:52.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:51 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:12:52.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:51 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:12:53.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:52 smithi173 ceph-mon[33839]: pgmap v756: 324 pgs: 324 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:12:53.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:12:53.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:52 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:53.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:52 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:53.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:52 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-94"}]: dispatch 2024-03-26T02:12:53.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:52 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-94"}]: dispatch 2024-03-26T02:12:53.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:52 smithi072 ceph-mon[30177]: pgmap v756: 324 pgs: 324 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:12:53.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:12:53.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:52 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:53.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:52 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:53.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:52 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-94"}]: dispatch 2024-03-26T02:12:53.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:52 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-94"}]: dispatch 2024-03-26T02:12:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:52 smithi072 ceph-mon[41619]: pgmap v756: 324 pgs: 324 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:12:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:12:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:52 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:52 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:12:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:52 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-94"}]: dispatch 2024-03-26T02:12:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:52 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-94"}]: dispatch 2024-03-26T02:12:55.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:54 smithi072 ceph-mon[30177]: pgmap v757: 324 pgs: 324 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:12:55.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:54 smithi072 ceph-mon[30177]: osdmap e494: 8 total, 8 up, 8 in 2024-03-26T02:12:55.220 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:12:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:12:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:12:55.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:54 smithi072 ceph-mon[41619]: pgmap v757: 324 pgs: 324 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:12:55.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:54 smithi072 ceph-mon[41619]: osdmap e494: 8 total, 8 up, 8 in 2024-03-26T02:12:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:54 smithi173 ceph-mon[33839]: pgmap v757: 324 pgs: 324 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:12:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:54 smithi173 ceph-mon[33839]: osdmap e494: 8 total, 8 up, 8 in 2024-03-26T02:12:56.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:55 smithi072 ceph-mon[30177]: osdmap e495: 8 total, 8 up, 8 in 2024-03-26T02:12:56.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:55 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:56.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:55 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:56.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:55 smithi072 ceph-mon[41619]: osdmap e495: 8 total, 8 up, 8 in 2024-03-26T02:12:56.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:55 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:56.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:55 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:55 smithi173 ceph-mon[33839]: osdmap e495: 8 total, 8 up, 8 in 2024-03-26T02:12:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:55 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:55 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:12:57.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:56 smithi173 ceph-mon[33839]: pgmap v760: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:12:57.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:56 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:12:57.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:56 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:12:57.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:56 smithi173 ceph-mon[33839]: osdmap e496: 8 total, 8 up, 8 in 2024-03-26T02:12:57.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:56 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:12:57.064 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:12:56 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=cleanup t=2024-03-26T02:12:56.636661011Z level=info msg="Completed cleanup jobs" duration=2.88123ms 2024-03-26T02:12:57.064 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:12:56 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=sqlstore.transactions t=2024-03-26T02:12:56.644428086Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T02:12:57.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:56 smithi072 ceph-mon[30177]: pgmap v760: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:12:57.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:56 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:12:57.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:56 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:12:57.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:56 smithi072 ceph-mon[30177]: osdmap e496: 8 total, 8 up, 8 in 2024-03-26T02:12:57.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:56 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:12:57.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:56 smithi072 ceph-mon[41619]: pgmap v760: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:12:57.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:56 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:12:57.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:56 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:12:57.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:56 smithi072 ceph-mon[41619]: osdmap e496: 8 total, 8 up, 8 in 2024-03-26T02:12:57.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:56 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:12:58.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:57 smithi072 ceph-mon[30177]: osdmap e497: 8 total, 8 up, 8 in 2024-03-26T02:12:58.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:57 smithi072 ceph-mon[41619]: osdmap e497: 8 total, 8 up, 8 in 2024-03-26T02:12:58.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:57 smithi173 ceph-mon[33839]: osdmap e497: 8 total, 8 up, 8 in 2024-03-26T02:12:59.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:12:58 smithi072 ceph-mon[30177]: pgmap v763: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:12:59.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:12:58 smithi072 ceph-mon[41619]: pgmap v763: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:12:59.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:12:58 smithi173 ceph-mon[33839]: pgmap v763: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:13:01.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:00 smithi072 ceph-mon[30177]: pgmap v764: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 875 B/s rd, 175 B/s wr, 1 op/s 2024-03-26T02:13:01.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:00 smithi072 ceph-mon[41619]: pgmap v764: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 875 B/s rd, 175 B/s wr, 1 op/s 2024-03-26T02:13:01.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:00 smithi173 ceph-mon[33839]: pgmap v764: 324 pgs: 32 creating+peering, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 875 B/s rd, 175 B/s wr, 1 op/s 2024-03-26T02:13:01.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:13:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:13:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:01 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:13:02.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:01 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:13:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:01 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:13:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:02 smithi072 ceph-mon[30177]: pgmap v765: 324 pgs: 324 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 898 B/s wr, 3 op/s 2024-03-26T02:13:03.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:02 smithi072 ceph-mon[41619]: pgmap v765: 324 pgs: 324 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 898 B/s wr, 3 op/s 2024-03-26T02:13:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:02 smithi173 ceph-mon[33839]: pgmap v765: 324 pgs: 324 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 898 B/s wr, 3 op/s 2024-03-26T02:13:05.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:04 smithi072 ceph-mon[30177]: pgmap v766: 324 pgs: 324 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 639 B/s wr, 1 op/s 2024-03-26T02:13:05.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:04 smithi072 ceph-mon[41619]: pgmap v766: 324 pgs: 324 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 639 B/s wr, 1 op/s 2024-03-26T02:13:05.220 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:13:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:13:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:13:05.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:04 smithi173 ceph-mon[33839]: pgmap v766: 324 pgs: 324 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 639 B/s wr, 1 op/s 2024-03-26T02:13:06.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:05 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:13:06.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:05 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:13:06.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:05 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:13:07.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:06 smithi072 ceph-mon[30177]: pgmap v767: 324 pgs: 324 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 520 B/s wr, 2 op/s 2024-03-26T02:13:07.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:06 smithi072 ceph-mon[41619]: pgmap v767: 324 pgs: 324 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 520 B/s wr, 2 op/s 2024-03-26T02:13:07.220 INFO:journalctl@ceph.osd.1.smithi072.stdout:Mar 26 02:13:06 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-1[52314]: 2024-03-26T02:13:06.865+0000 7f398e848700 -1 snap_mapper.add_oid found existing snaps mapped on 108:70a1cda1:test-rados-api-smithi072-82692-97::foo:21, removing 2024-03-26T02:13:07.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:06 smithi173 ceph-mon[33839]: pgmap v767: 324 pgs: 324 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 520 B/s wr, 2 op/s 2024-03-26T02:13:07.313 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:13:06 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:13:06.865+0000 7f2dd9e2a700 -1 snap_mapper.add_oid found existing snaps mapped on 108:70a1cda1:test-rados-api-smithi072-82692-97::foo:21, removing 2024-03-26T02:13:07.314 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 02:13:06 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-4[39374]: 2024-03-26T02:13:06.865+0000 7fc2d95ea700 -1 snap_mapper.add_oid found existing snaps mapped on 108:70a1cda1:test-rados-api-smithi072-82692-97::foo:21, removing 2024-03-26T02:13:08.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:13:08.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:07 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:13:08.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-96"}]: dispatch 2024-03-26T02:13:08.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:07 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-96"}]: dispatch 2024-03-26T02:13:08.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:07 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:13:08.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:13:08.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:07 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:13:08.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-96"}]: dispatch 2024-03-26T02:13:08.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:07 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-96"}]: dispatch 2024-03-26T02:13:08.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:07 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:13:08.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:13:08.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:07 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:13:08.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-96"}]: dispatch 2024-03-26T02:13:08.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:07 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-96"}]: dispatch 2024-03-26T02:13:08.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:07 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:13:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:08 smithi072 ceph-mon[30177]: pgmap v768: 324 pgs: 324 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 945 B/s rd, 472 B/s wr, 1 op/s 2024-03-26T02:13:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:08 smithi072 ceph-mon[30177]: osdmap e498: 8 total, 8 up, 8 in 2024-03-26T02:13:09.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:08 smithi072 ceph-mon[41619]: pgmap v768: 324 pgs: 324 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 945 B/s rd, 472 B/s wr, 1 op/s 2024-03-26T02:13:09.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:08 smithi072 ceph-mon[41619]: osdmap e498: 8 total, 8 up, 8 in 2024-03-26T02:13:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:08 smithi173 ceph-mon[33839]: pgmap v768: 324 pgs: 324 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 945 B/s rd, 472 B/s wr, 1 op/s 2024-03-26T02:13:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:08 smithi173 ceph-mon[33839]: osdmap e498: 8 total, 8 up, 8 in 2024-03-26T02:13:10.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:09 smithi072 ceph-mon[30177]: osdmap e499: 8 total, 8 up, 8 in 2024-03-26T02:13:10.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:10.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:09 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:10.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:09 smithi072 ceph-mon[41619]: osdmap e499: 8 total, 8 up, 8 in 2024-03-26T02:13:10.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:10.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:09 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:10.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:09 smithi173 ceph-mon[33839]: osdmap e499: 8 total, 8 up, 8 in 2024-03-26T02:13:10.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:10.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:09 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:11.215 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:13:11 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:13:11.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:10 smithi173 ceph-mon[33839]: pgmap v771: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:13:11.216 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:13:11.216 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:10 smithi173 ceph-mon[33839]: osdmap e500: 8 total, 8 up, 8 in 2024-03-26T02:13:11.216 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:13:11.216 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:10 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:13:11.216 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:10 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:13:11.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:10 smithi072 ceph-mon[30177]: pgmap v771: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:13:11.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:13:11.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:10 smithi072 ceph-mon[30177]: osdmap e500: 8 total, 8 up, 8 in 2024-03-26T02:13:11.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:13:11.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:10 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:13:11.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:10 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:13:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:10 smithi072 ceph-mon[41619]: pgmap v771: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:13:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:13:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:10 smithi072 ceph-mon[41619]: osdmap e500: 8 total, 8 up, 8 in 2024-03-26T02:13:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:13:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:10 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:13:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:10 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:13:12.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:11 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:13:12.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:11 smithi072 ceph-mon[30177]: osdmap e501: 8 total, 8 up, 8 in 2024-03-26T02:13:12.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:11 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-98", "mode": "writeback"}]: dispatch 2024-03-26T02:13:12.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:11 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-98", "mode": "writeback"}]: dispatch 2024-03-26T02:13:12.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:11 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:13:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:11 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:13:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:11 smithi072 ceph-mon[41619]: osdmap e501: 8 total, 8 up, 8 in 2024-03-26T02:13:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:11 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-98", "mode": "writeback"}]: dispatch 2024-03-26T02:13:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:11 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-98", "mode": "writeback"}]: dispatch 2024-03-26T02:13:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:11 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:13:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:11 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:13:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:11 smithi173 ceph-mon[33839]: osdmap e501: 8 total, 8 up, 8 in 2024-03-26T02:13:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:11 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-98", "mode": "writeback"}]: dispatch 2024-03-26T02:13:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:11 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-98", "mode": "writeback"}]: dispatch 2024-03-26T02:13:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:11 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:13:13.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:12 smithi072 ceph-mon[30177]: pgmap v774: 324 pgs: 324 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-03-26T02:13:13.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:12 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:13:13.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-98", "mode": "writeback"}]': finished 2024-03-26T02:13:13.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:12 smithi072 ceph-mon[30177]: osdmap e502: 8 total, 8 up, 8 in 2024-03-26T02:13:13.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-98"}]: dispatch 2024-03-26T02:13:13.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-98"}]: dispatch 2024-03-26T02:13:13.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:12 smithi072 ceph-mon[41619]: pgmap v774: 324 pgs: 324 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-03-26T02:13:13.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:12 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:13:13.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-98", "mode": "writeback"}]': finished 2024-03-26T02:13:13.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:12 smithi072 ceph-mon[41619]: osdmap e502: 8 total, 8 up, 8 in 2024-03-26T02:13:13.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-98"}]: dispatch 2024-03-26T02:13:13.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-98"}]: dispatch 2024-03-26T02:13:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:12 smithi173 ceph-mon[33839]: pgmap v774: 324 pgs: 324 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-03-26T02:13:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:12 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:13:13.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-98", "mode": "writeback"}]': finished 2024-03-26T02:13:13.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:12 smithi173 ceph-mon[33839]: osdmap e502: 8 total, 8 up, 8 in 2024-03-26T02:13:13.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-98"}]: dispatch 2024-03-26T02:13:13.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-98"}]: dispatch 2024-03-26T02:13:14.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:13 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-98"}]': finished 2024-03-26T02:13:14.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:13 smithi072 ceph-mon[30177]: osdmap e503: 8 total, 8 up, 8 in 2024-03-26T02:13:14.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:13 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:13:14.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:13 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:13:14.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:13 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-98"}]': finished 2024-03-26T02:13:14.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:13 smithi072 ceph-mon[41619]: osdmap e503: 8 total, 8 up, 8 in 2024-03-26T02:13:14.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:13 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:13:14.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:13 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:13:14.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:13 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-98"}]': finished 2024-03-26T02:13:14.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:13 smithi173 ceph-mon[33839]: osdmap e503: 8 total, 8 up, 8 in 2024-03-26T02:13:14.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:13 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:13:14.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:13 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:13:15.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:13:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:13:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:13:15.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:14 smithi072 ceph-mon[30177]: pgmap v777: 324 pgs: 324 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T02:13:15.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:14 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:13:15.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:14 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T02:13:15.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:14 smithi072 ceph-mon[30177]: osdmap e504: 8 total, 8 up, 8 in 2024-03-26T02:13:15.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:14 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T02:13:15.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:14 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T02:13:15.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:14 smithi072 ceph-mon[41619]: pgmap v777: 324 pgs: 324 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T02:13:15.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:14 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:13:15.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:14 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T02:13:15.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:14 smithi072 ceph-mon[41619]: osdmap e504: 8 total, 8 up, 8 in 2024-03-26T02:13:15.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:14 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T02:13:15.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:14 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T02:13:15.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:14 smithi173 ceph-mon[33839]: pgmap v777: 324 pgs: 324 active+clean; 8.4 MiB data, 623 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T02:13:15.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:14 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:13:15.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:14 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T02:13:15.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:14 smithi173 ceph-mon[33839]: osdmap e504: 8 total, 8 up, 8 in 2024-03-26T02:13:15.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:14 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T02:13:15.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:14 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T02:13:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_count","val": "1"}]': finished 2024-03-26T02:13:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:15 smithi072 ceph-mon[30177]: osdmap e505: 8 total, 8 up, 8 in 2024-03-26T02:13:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:13:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:13:16.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:15 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:13:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_count","val": "1"}]': finished 2024-03-26T02:13:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:15 smithi072 ceph-mon[41619]: osdmap e505: 8 total, 8 up, 8 in 2024-03-26T02:13:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:13:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:13:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:15 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:13:16.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_count","val": "1"}]': finished 2024-03-26T02:13:16.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:15 smithi173 ceph-mon[33839]: osdmap e505: 8 total, 8 up, 8 in 2024-03-26T02:13:16.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:13:16.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:13:16.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:15 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:13:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:16 smithi072 ceph-mon[30177]: pgmap v780: 324 pgs: 324 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:16 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:13:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:16 smithi072 ceph-mon[30177]: osdmap e506: 8 total, 8 up, 8 in 2024-03-26T02:13:17.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:16 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T02:13:17.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:16 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T02:13:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:16 smithi072 ceph-mon[41619]: pgmap v780: 324 pgs: 324 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:16 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:13:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:16 smithi072 ceph-mon[41619]: osdmap e506: 8 total, 8 up, 8 in 2024-03-26T02:13:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:16 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T02:13:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:16 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T02:13:17.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:16 smithi173 ceph-mon[33839]: pgmap v780: 324 pgs: 324 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:17.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:16 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:13:17.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:16 smithi173 ceph-mon[33839]: osdmap e506: 8 total, 8 up, 8 in 2024-03-26T02:13:17.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:16 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T02:13:17.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:16 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T02:13:18.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:17 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "target_max_objects","val": "250"}]': finished 2024-03-26T02:13:18.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:17 smithi072 ceph-mon[30177]: osdmap e507: 8 total, 8 up, 8 in 2024-03-26T02:13:18.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:17 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:13:18.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:17 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:13:18.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:17 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "target_max_objects","val": "250"}]': finished 2024-03-26T02:13:18.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:17 smithi072 ceph-mon[41619]: osdmap e507: 8 total, 8 up, 8 in 2024-03-26T02:13:18.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:17 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:13:18.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:17 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:13:18.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:17 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-98","var": "target_max_objects","val": "250"}]': finished 2024-03-26T02:13:18.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:17 smithi173 ceph-mon[33839]: osdmap e507: 8 total, 8 up, 8 in 2024-03-26T02:13:18.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:17 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:13:18.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:17 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:13:19.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:18 smithi072 ceph-mon[30177]: pgmap v783: 324 pgs: 324 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:19.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:13:19.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:18 smithi072 ceph-mon[30177]: osdmap e508: 8 total, 8 up, 8 in 2024-03-26T02:13:19.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-98"}]: dispatch 2024-03-26T02:13:19.222 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:18 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-98"}]: dispatch 2024-03-26T02:13:19.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:18 smithi072 ceph-mon[41619]: pgmap v783: 324 pgs: 324 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:19.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:13:19.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:18 smithi072 ceph-mon[41619]: osdmap e508: 8 total, 8 up, 8 in 2024-03-26T02:13:19.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-98"}]: dispatch 2024-03-26T02:13:19.223 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:18 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-98"}]: dispatch 2024-03-26T02:13:19.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:18 smithi173 ceph-mon[33839]: pgmap v783: 324 pgs: 324 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:19.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:13:19.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:18 smithi173 ceph-mon[33839]: osdmap e508: 8 total, 8 up, 8 in 2024-03-26T02:13:19.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-98"}]: dispatch 2024-03-26T02:13:19.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:18 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-98"}]: dispatch 2024-03-26T02:13:20.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:19 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-98"}]': finished 2024-03-26T02:13:20.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:19 smithi072 ceph-mon[30177]: osdmap e509: 8 total, 8 up, 8 in 2024-03-26T02:13:20.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:19 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-98"}]': finished 2024-03-26T02:13:20.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:19 smithi072 ceph-mon[41619]: osdmap e509: 8 total, 8 up, 8 in 2024-03-26T02:13:20.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:19 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-98"}]': finished 2024-03-26T02:13:20.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:19 smithi173 ceph-mon[33839]: osdmap e509: 8 total, 8 up, 8 in 2024-03-26T02:13:21.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:20 smithi072 ceph-mon[30177]: pgmap v786: 324 pgs: 324 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:13:21.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:20 smithi072 ceph-mon[30177]: osdmap e510: 8 total, 8 up, 8 in 2024-03-26T02:13:21.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:20 smithi072 ceph-mon[41619]: pgmap v786: 324 pgs: 324 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:13:21.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:20 smithi072 ceph-mon[41619]: osdmap e510: 8 total, 8 up, 8 in 2024-03-26T02:13:21.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:13:21 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:13:21.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:20 smithi173 ceph-mon[33839]: pgmap v786: 324 pgs: 324 active+clean; 8.4 MiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:13:21.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:20 smithi173 ceph-mon[33839]: osdmap e510: 8 total, 8 up, 8 in 2024-03-26T02:13:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:21 smithi173 ceph-mon[33839]: osdmap e511: 8 total, 8 up, 8 in 2024-03-26T02:13:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:21 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:22.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:21 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:13:22.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:21 smithi072 ceph-mon[30177]: osdmap e511: 8 total, 8 up, 8 in 2024-03-26T02:13:22.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:22.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:21 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:22.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:21 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:13:22.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:21 smithi072 ceph-mon[41619]: osdmap e511: 8 total, 8 up, 8 in 2024-03-26T02:13:22.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:22.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:21 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:22.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:21 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:13:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:22 smithi173 ceph-mon[33839]: pgmap v789: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:22 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:13:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:22 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:13:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:22 smithi173 ceph-mon[33839]: osdmap e512: 8 total, 8 up, 8 in 2024-03-26T02:13:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:22 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:13:23.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:22 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:13:23.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:22 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:13:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:22 smithi072 ceph-mon[30177]: pgmap v789: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:22 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:13:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:22 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:13:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:22 smithi072 ceph-mon[30177]: osdmap e512: 8 total, 8 up, 8 in 2024-03-26T02:13:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:22 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:13:23.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:22 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:13:23.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:22 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:13:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:22 smithi072 ceph-mon[41619]: pgmap v789: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:22 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:13:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:22 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:13:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:22 smithi072 ceph-mon[41619]: osdmap e512: 8 total, 8 up, 8 in 2024-03-26T02:13:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:22 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:13:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:22 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:13:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:22 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:13:24.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:23 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:13:24.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:23 smithi173 ceph-mon[33839]: osdmap e513: 8 total, 8 up, 8 in 2024-03-26T02:13:24.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:23 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-100"}]: dispatch 2024-03-26T02:13:24.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:23 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-100"}]: dispatch 2024-03-26T02:13:24.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:23 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:13:24.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:23 smithi072 ceph-mon[30177]: osdmap e513: 8 total, 8 up, 8 in 2024-03-26T02:13:24.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:23 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-100"}]: dispatch 2024-03-26T02:13:24.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:23 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-100"}]: dispatch 2024-03-26T02:13:24.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:23 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:13:24.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:23 smithi072 ceph-mon[41619]: osdmap e513: 8 total, 8 up, 8 in 2024-03-26T02:13:24.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:23 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-100"}]: dispatch 2024-03-26T02:13:24.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:23 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-100"}]: dispatch 2024-03-26T02:13:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:25 smithi173 ceph-mon[33839]: pgmap v792: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:25 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-100"}]': finished 2024-03-26T02:13:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:25 smithi173 ceph-mon[33839]: osdmap e514: 8 total, 8 up, 8 in 2024-03-26T02:13:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-100", "mode": "writeback"}]: dispatch 2024-03-26T02:13:25.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:25 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-100", "mode": "writeback"}]: dispatch 2024-03-26T02:13:25.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:13:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:13:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:13:25.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:25 smithi072 ceph-mon[30177]: pgmap v792: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:25.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:25 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-100"}]': finished 2024-03-26T02:13:25.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:25 smithi072 ceph-mon[30177]: osdmap e514: 8 total, 8 up, 8 in 2024-03-26T02:13:25.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-100", "mode": "writeback"}]: dispatch 2024-03-26T02:13:25.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:25 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-100", "mode": "writeback"}]: dispatch 2024-03-26T02:13:25.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:25 smithi072 ceph-mon[41619]: pgmap v792: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:25.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:25 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-6", "overlaypool": "test-rados-api-smithi072-82692-100"}]': finished 2024-03-26T02:13:25.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:25 smithi072 ceph-mon[41619]: osdmap e514: 8 total, 8 up, 8 in 2024-03-26T02:13:25.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-100", "mode": "writeback"}]: dispatch 2024-03-26T02:13:25.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:25 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-100", "mode": "writeback"}]: dispatch 2024-03-26T02:13:26.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:26 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:13:26.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:26 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-100", "mode": "writeback"}]': finished 2024-03-26T02:13:26.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:26 smithi173 ceph-mon[33839]: osdmap e515: 8 total, 8 up, 8 in 2024-03-26T02:13:26.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:26 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:13:26.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:26 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:13:26.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:26 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_count","val": "2"}]': finished 2024-03-26T02:13:26.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:26 smithi173 ceph-mon[33839]: osdmap e516: 8 total, 8 up, 8 in 2024-03-26T02:13:26.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:26 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:13:26.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:26 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-100", "mode": "writeback"}]': finished 2024-03-26T02:13:26.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:26 smithi072 ceph-mon[30177]: osdmap e515: 8 total, 8 up, 8 in 2024-03-26T02:13:26.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:26 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:13:26.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:26 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:13:26.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:26 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_count","val": "2"}]': finished 2024-03-26T02:13:26.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:26 smithi072 ceph-mon[30177]: osdmap e516: 8 total, 8 up, 8 in 2024-03-26T02:13:26.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:26 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:13:26.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:26 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-100", "mode": "writeback"}]': finished 2024-03-26T02:13:26.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:26 smithi072 ceph-mon[41619]: osdmap e515: 8 total, 8 up, 8 in 2024-03-26T02:13:26.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:26 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:13:26.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:26 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:13:26.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:26 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_count","val": "2"}]': finished 2024-03-26T02:13:26.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:26 smithi072 ceph-mon[41619]: osdmap e516: 8 total, 8 up, 8 in 2024-03-26T02:13:27.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:27 smithi173 ceph-mon[33839]: pgmap v795: 324 pgs: 324 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:13:27.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:27 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:13:27.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:27 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:13:27.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:27 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:13:27.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:27 smithi173 ceph-mon[33839]: osdmap e517: 8 total, 8 up, 8 in 2024-03-26T02:13:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:27 smithi072 ceph-mon[30177]: pgmap v795: 324 pgs: 324 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:13:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:27 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:13:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:27 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:13:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:27 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:13:27.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:27 smithi072 ceph-mon[30177]: osdmap e517: 8 total, 8 up, 8 in 2024-03-26T02:13:27.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:27 smithi072 ceph-mon[41619]: pgmap v795: 324 pgs: 324 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:13:27.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:27 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:13:27.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:27 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:13:27.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:27 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:13:27.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:27 smithi072 ceph-mon[41619]: osdmap e517: 8 total, 8 up, 8 in 2024-03-26T02:13:28.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:28 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T02:13:28.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:28 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T02:13:28.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:28 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T02:13:28.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:28 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T02:13:28.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:28 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T02:13:28.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:28 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T02:13:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:29 smithi173 ceph-mon[33839]: pgmap v798: 324 pgs: 324 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:13:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:29 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:13:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:29 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T02:13:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:29 smithi173 ceph-mon[33839]: osdmap e518: 8 total, 8 up, 8 in 2024-03-26T02:13:29.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:29 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T02:13:29.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:29 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T02:13:29.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:29 smithi072 ceph-mon[30177]: pgmap v798: 324 pgs: 324 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:13:29.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:29 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:13:29.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:29 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T02:13:29.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:29 smithi072 ceph-mon[30177]: osdmap e518: 8 total, 8 up, 8 in 2024-03-26T02:13:29.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:29 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T02:13:29.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:29 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T02:13:29.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:29 smithi072 ceph-mon[41619]: pgmap v798: 324 pgs: 324 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:13:29.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:29 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:13:29.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:29 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T02:13:29.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:29 smithi072 ceph-mon[41619]: osdmap e518: 8 total, 8 up, 8 in 2024-03-26T02:13:29.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:29 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T02:13:29.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:29 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T02:13:30.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-26T02:13:30.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:30 smithi173 ceph-mon[33839]: osdmap e519: 8 total, 8 up, 8 in 2024-03-26T02:13:30.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:13:30.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:13:30.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-26T02:13:30.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:30 smithi072 ceph-mon[30177]: osdmap e519: 8 total, 8 up, 8 in 2024-03-26T02:13:30.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:13:30.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:13:30.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-26T02:13:30.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:30 smithi072 ceph-mon[41619]: osdmap e519: 8 total, 8 up, 8 in 2024-03-26T02:13:30.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:13:30.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:13:31.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:13:31 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:13:31.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:31 smithi173 ceph-mon[33839]: pgmap v801: 324 pgs: 324 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:13:31.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:31 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:13:31.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:31 smithi173 ceph-mon[33839]: osdmap e520: 8 total, 8 up, 8 in 2024-03-26T02:13:31.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:31 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-100"}]: dispatch 2024-03-26T02:13:31.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:31 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-100"}]: dispatch 2024-03-26T02:13:31.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:31 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:13:31.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:31 smithi072 ceph-mon[41619]: pgmap v801: 324 pgs: 324 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:13:31.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:31 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:13:31.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:31 smithi072 ceph-mon[41619]: osdmap e520: 8 total, 8 up, 8 in 2024-03-26T02:13:31.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:31 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-100"}]: dispatch 2024-03-26T02:13:31.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:31 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-100"}]: dispatch 2024-03-26T02:13:31.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:31 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:13:31.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:31 smithi072 ceph-mon[30177]: pgmap v801: 324 pgs: 324 active+clean; 8.4 MiB data, 625 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:13:31.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:31 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]': finished 2024-03-26T02:13:31.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:31 smithi072 ceph-mon[30177]: osdmap e520: 8 total, 8 up, 8 in 2024-03-26T02:13:31.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:31 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-100"}]: dispatch 2024-03-26T02:13:31.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:31 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-100"}]: dispatch 2024-03-26T02:13:31.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:31 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:13:32.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:32 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-100"}]': finished 2024-03-26T02:13:32.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:32 smithi173 ceph-mon[33839]: osdmap e521: 8 total, 8 up, 8 in 2024-03-26T02:13:32.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:32 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:13:32.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:32 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-100"}]': finished 2024-03-26T02:13:32.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:32 smithi072 ceph-mon[30177]: osdmap e521: 8 total, 8 up, 8 in 2024-03-26T02:13:32.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:32 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:13:32.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:32 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-100"}]': finished 2024-03-26T02:13:32.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:32 smithi072 ceph-mon[41619]: osdmap e521: 8 total, 8 up, 8 in 2024-03-26T02:13:32.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:32 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:13:33.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:33 smithi173 ceph-mon[33839]: pgmap v804: 324 pgs: 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T02:13:33.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:33 smithi173 ceph-mon[33839]: osdmap e522: 8 total, 8 up, 8 in 2024-03-26T02:13:33.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:33 smithi072 ceph-mon[30177]: pgmap v804: 324 pgs: 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T02:13:33.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:33 smithi072 ceph-mon[30177]: osdmap e522: 8 total, 8 up, 8 in 2024-03-26T02:13:33.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:33 smithi072 ceph-mon[41619]: pgmap v804: 324 pgs: 324 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T02:13:33.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:33 smithi072 ceph-mon[41619]: osdmap e522: 8 total, 8 up, 8 in 2024-03-26T02:13:34.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:34 smithi072 ceph-mon[30177]: osdmap e523: 8 total, 8 up, 8 in 2024-03-26T02:13:34.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:34 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:34.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:34 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:34.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:34 smithi072 ceph-mon[41619]: osdmap e523: 8 total, 8 up, 8 in 2024-03-26T02:13:34.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:34 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:34.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:34 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:34.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:34 smithi173 ceph-mon[33839]: osdmap e523: 8 total, 8 up, 8 in 2024-03-26T02:13:34.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:34 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:34.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:34 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:35.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:13:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:13:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:13:35.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:35 smithi072 ceph-mon[30177]: pgmap v807: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:13:35.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:35 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:13:35.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:35 smithi072 ceph-mon[30177]: osdmap e524: 8 total, 8 up, 8 in 2024-03-26T02:13:35.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:35 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:13:35.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:35 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:13:35.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:35 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:13:35.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:35 smithi072 ceph-mon[41619]: pgmap v807: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:13:35.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:35 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:13:35.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:35 smithi072 ceph-mon[41619]: osdmap e524: 8 total, 8 up, 8 in 2024-03-26T02:13:35.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:35 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:13:35.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:35 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:13:35.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:35 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:13:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:35 smithi173 ceph-mon[33839]: pgmap v807: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:13:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:35 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:13:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:35 smithi173 ceph-mon[33839]: osdmap e524: 8 total, 8 up, 8 in 2024-03-26T02:13:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:35 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T02:13:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:35 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:13:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:35 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T02:13:36.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:36 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:13:36.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:36 smithi072 ceph-mon[41619]: osdmap e525: 8 total, 8 up, 8 in 2024-03-26T02:13:36.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:36 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:13:36.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:36 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:13:36.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:36 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:13:36.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:36 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T02:13:36.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:36 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:13:36.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:36 smithi072 ceph-mon[30177]: osdmap e525: 8 total, 8 up, 8 in 2024-03-26T02:13:36.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:36 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:13:36.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:36 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:13:36.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:36 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:13:36.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:36 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T02:13:36.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:36 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T02:13:36.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:36 smithi173 ceph-mon[33839]: osdmap e525: 8 total, 8 up, 8 in 2024-03-26T02:13:36.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:36 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:13:36.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:36 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T02:13:36.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:36 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:13:36.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:36 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T02:13:37.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:37 smithi072 ceph-mon[30177]: pgmap v810: 324 pgs: 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:37.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:37 smithi072 ceph-mon[30177]: osdmap e526: 8 total, 8 up, 8 in 2024-03-26T02:13:37.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:37 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:13:37.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:37 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:13:37.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:37 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T02:13:37.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:37 smithi072 ceph-mon[30177]: osdmap e527: 8 total, 8 up, 8 in 2024-03-26T02:13:37.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:37 smithi072 ceph-mon[41619]: pgmap v810: 324 pgs: 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:37.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:37 smithi072 ceph-mon[41619]: osdmap e526: 8 total, 8 up, 8 in 2024-03-26T02:13:37.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:37 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:13:37.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:37 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:13:37.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:37 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T02:13:37.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:37 smithi072 ceph-mon[41619]: osdmap e527: 8 total, 8 up, 8 in 2024-03-26T02:13:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:37 smithi173 ceph-mon[33839]: pgmap v810: 324 pgs: 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:37 smithi173 ceph-mon[33839]: osdmap e526: 8 total, 8 up, 8 in 2024-03-26T02:13:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:37 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:13:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:37 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T02:13:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:37 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T02:13:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:37 smithi173 ceph-mon[33839]: osdmap e527: 8 total, 8 up, 8 in 2024-03-26T02:13:38.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:38 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:13:38.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:38 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:13:38.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:38 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-102"}]: dispatch 2024-03-26T02:13:38.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:38 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-102"}]: dispatch 2024-03-26T02:13:38.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:38 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:13:38.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:38 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:13:38.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:38 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:13:38.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:38 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-102"}]: dispatch 2024-03-26T02:13:38.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:38 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-102"}]: dispatch 2024-03-26T02:13:38.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:38 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:13:38.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:38 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:13:38.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:38 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-6"}]: dispatch 2024-03-26T02:13:38.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:38 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2677969943' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-102"}]: dispatch 2024-03-26T02:13:38.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:38 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-6", "tierpool": "test-rados-api-smithi072-82692-102"}]: dispatch 2024-03-26T02:13:38.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:38 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:13:39.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:39 smithi072 ceph-mon[30177]: pgmap v813: 324 pgs: 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:39.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:39 smithi072 ceph-mon[30177]: osdmap e528: 8 total, 8 up, 8 in 2024-03-26T02:13:39.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:39 smithi072 ceph-mon[41619]: pgmap v813: 324 pgs: 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:39.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:39 smithi072 ceph-mon[41619]: osdmap e528: 8 total, 8 up, 8 in 2024-03-26T02:13:39.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:39 smithi173 ceph-mon[33839]: pgmap v813: 324 pgs: 324 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:39.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:39 smithi173 ceph-mon[33839]: osdmap e528: 8 total, 8 up, 8 in 2024-03-26T02:13:40.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:40 smithi072 ceph-mon[30177]: osdmap e529: 8 total, 8 up, 8 in 2024-03-26T02:13:40.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi072-82692-104"}]: dispatch 2024-03-26T02:13:40.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi072-82692-104"}]: dispatch 2024-03-26T02:13:40.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi072-82692-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:13:40.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:40 smithi072 ceph-mon[41619]: osdmap e529: 8 total, 8 up, 8 in 2024-03-26T02:13:40.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi072-82692-104"}]: dispatch 2024-03-26T02:13:40.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi072-82692-104"}]: dispatch 2024-03-26T02:13:40.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi072-82692-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:13:40.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:40 smithi173 ceph-mon[33839]: osdmap e529: 8 total, 8 up, 8 in 2024-03-26T02:13:40.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi072-82692-104"}]: dispatch 2024-03-26T02:13:40.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi072-82692-104"}]: dispatch 2024-03-26T02:13:40.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi072-82692-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:13:41.370 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:41 smithi072 ceph-mon[30177]: pgmap v816: 260 pgs: 260 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:13:41.370 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:41 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi072-82692-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:13:41.371 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:41 smithi072 ceph-mon[30177]: osdmap e530: 8 total, 8 up, 8 in 2024-03-26T02:13:41.371 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:41 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi072-82692-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi072-82692-104"}]: dispatch 2024-03-26T02:13:41.371 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:41 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:13:41.371 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:41 smithi072 ceph-mon[41619]: pgmap v816: 260 pgs: 260 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:13:41.371 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:41 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi072-82692-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:13:41.371 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:41 smithi072 ceph-mon[41619]: osdmap e530: 8 total, 8 up, 8 in 2024-03-26T02:13:41.371 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:41 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi072-82692-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi072-82692-104"}]: dispatch 2024-03-26T02:13:41.371 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:41 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:13:41.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:13:41 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:13:41.465 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:41 smithi173 ceph-mon[33839]: pgmap v816: 260 pgs: 260 active+clean; 582 KiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:13:41.465 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:41 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi072-82692-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:13:41.465 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:41 smithi173 ceph-mon[33839]: osdmap e530: 8 total, 8 up, 8 in 2024-03-26T02:13:41.465 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:41 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi072-82692-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi072-82692-104"}]: dispatch 2024-03-26T02:13:41.465 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:41 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:13:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:42 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:13:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:42 smithi072 ceph-mon[30177]: osdmap e531: 8 total, 8 up, 8 in 2024-03-26T02:13:42.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:42 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:13:42.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:42 smithi072 ceph-mon[41619]: osdmap e531: 8 total, 8 up, 8 in 2024-03-26T02:13:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:42 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:13:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:42 smithi173 ceph-mon[33839]: osdmap e531: 8 total, 8 up, 8 in 2024-03-26T02:13:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:43 smithi072 ceph-mon[30177]: pgmap v819: 260 pgs: 260 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:13:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:43 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi072-82692-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi072-82692-104"}]': finished 2024-03-26T02:13:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:43 smithi072 ceph-mon[30177]: osdmap e532: 8 total, 8 up, 8 in 2024-03-26T02:13:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:43 smithi072 ceph-mon[41619]: pgmap v819: 260 pgs: 260 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:13:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:43 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi072-82692-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi072-82692-104"}]': finished 2024-03-26T02:13:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:43 smithi072 ceph-mon[41619]: osdmap e532: 8 total, 8 up, 8 in 2024-03-26T02:13:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:43 smithi173 ceph-mon[33839]: pgmap v819: 260 pgs: 260 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:13:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:43 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi072-82692-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi072-82692-104"}]': finished 2024-03-26T02:13:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:43 smithi173 ceph-mon[33839]: osdmap e532: 8 total, 8 up, 8 in 2024-03-26T02:13:44.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:44 smithi072 ceph-mon[30177]: osdmap e533: 8 total, 8 up, 8 in 2024-03-26T02:13:44.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:44 smithi072 ceph-mon[41619]: osdmap e533: 8 total, 8 up, 8 in 2024-03-26T02:13:44.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:44 smithi173 ceph-mon[33839]: osdmap e533: 8 total, 8 up, 8 in 2024-03-26T02:13:45.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:13:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:13:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:13:45.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:45 smithi072 ceph-mon[30177]: pgmap v822: 268 pgs: 8 unknown, 260 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:45.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:45 smithi072 ceph-mon[30177]: osdmap e534: 8 total, 8 up, 8 in 2024-03-26T02:13:45.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:45 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2374952337' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:45.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:45 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:45.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:45 smithi072 ceph-mon[41619]: pgmap v822: 268 pgs: 8 unknown, 260 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:45.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:45 smithi072 ceph-mon[41619]: osdmap e534: 8 total, 8 up, 8 in 2024-03-26T02:13:45.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:45 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2374952337' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:45.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:45 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:45 smithi173 ceph-mon[33839]: pgmap v822: 268 pgs: 8 unknown, 260 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:45 smithi173 ceph-mon[33839]: osdmap e534: 8 total, 8 up, 8 in 2024-03-26T02:13:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:45 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2374952337' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:45.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:45 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:46.150 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:46 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:13:46.150 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:46 smithi072 ceph-mon[41619]: osdmap e535: 8 total, 8 up, 8 in 2024-03-26T02:13:46.151 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:46 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:13:46.151 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:46 smithi072 ceph-mon[30177]: osdmap e535: 8 total, 8 up, 8 in 2024-03-26T02:13:46.151 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:46 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:13:46.151 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:46 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:13:46.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:46 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:13:46.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:46 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:13:46.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:46 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:13:46.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:46 smithi173 ceph-mon[33839]: osdmap e535: 8 total, 8 up, 8 in 2024-03-26T02:13:46.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:46 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:13:46.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:46 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:13:47.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:47 smithi072 ceph-mon[41619]: pgmap v825: 300 pgs: 300 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:13:47.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:47 smithi072 ceph-mon[41619]: osdmap e536: 8 total, 8 up, 8 in 2024-03-26T02:13:47.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:47 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2374952337' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:47.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:47 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:47.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:47 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:13:47.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:47 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:13:47.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:47 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:13:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:47 smithi072 ceph-mon[30177]: pgmap v825: 300 pgs: 300 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:13:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:47 smithi072 ceph-mon[30177]: osdmap e536: 8 total, 8 up, 8 in 2024-03-26T02:13:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:47 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2374952337' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:47 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:47 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:13:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:47 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:13:47.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:47 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:13:47.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:47 smithi173 ceph-mon[33839]: pgmap v825: 300 pgs: 300 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:13:47.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:47 smithi173 ceph-mon[33839]: osdmap e536: 8 total, 8 up, 8 in 2024-03-26T02:13:47.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:47 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2374952337' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:47.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:47 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:47.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:47 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:13:47.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:47 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:13:47.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:47 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:13:48.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:48 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:13:48.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:48 smithi072 ceph-mon[30177]: osdmap e537: 8 total, 8 up, 8 in 2024-03-26T02:13:48.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:48 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2374952337' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-107", "tierpool": "test-rados-api-smithi072-82692-107-cache"}]: dispatch 2024-03-26T02:13:48.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:48 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-107", "tierpool": "test-rados-api-smithi072-82692-107-cache"}]: dispatch 2024-03-26T02:13:48.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:48 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:13:48.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:48 smithi072 ceph-mon[41619]: osdmap e537: 8 total, 8 up, 8 in 2024-03-26T02:13:48.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:48 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2374952337' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-107", "tierpool": "test-rados-api-smithi072-82692-107-cache"}]: dispatch 2024-03-26T02:13:48.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:48 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-107", "tierpool": "test-rados-api-smithi072-82692-107-cache"}]: dispatch 2024-03-26T02:13:48.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:48 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:13:48.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:48 smithi173 ceph-mon[33839]: osdmap e537: 8 total, 8 up, 8 in 2024-03-26T02:13:48.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:48 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2374952337' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-107", "tierpool": "test-rados-api-smithi072-82692-107-cache"}]: dispatch 2024-03-26T02:13:48.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:48 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-107", "tierpool": "test-rados-api-smithi072-82692-107-cache"}]: dispatch 2024-03-26T02:13:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:49 smithi072 ceph-mon[30177]: pgmap v828: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:13:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:49 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-107", "tierpool": "test-rados-api-smithi072-82692-107-cache"}]': finished 2024-03-26T02:13:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:49 smithi072 ceph-mon[30177]: osdmap e538: 8 total, 8 up, 8 in 2024-03-26T02:13:49.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:49 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2374952337' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-107", "overlaypool": "test-rados-api-smithi072-82692-107-cache"}]: dispatch 2024-03-26T02:13:49.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:49 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-107", "overlaypool": "test-rados-api-smithi072-82692-107-cache"}]: dispatch 2024-03-26T02:13:49.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:49 smithi072 ceph-mon[41619]: pgmap v828: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:13:49.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:49 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-107", "tierpool": "test-rados-api-smithi072-82692-107-cache"}]': finished 2024-03-26T02:13:49.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:49 smithi072 ceph-mon[41619]: osdmap e538: 8 total, 8 up, 8 in 2024-03-26T02:13:49.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:49 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2374952337' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-107", "overlaypool": "test-rados-api-smithi072-82692-107-cache"}]: dispatch 2024-03-26T02:13:49.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:49 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-107", "overlaypool": "test-rados-api-smithi072-82692-107-cache"}]: dispatch 2024-03-26T02:13:49.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:49 smithi173 ceph-mon[33839]: pgmap v828: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:13:49.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:49 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-107", "tierpool": "test-rados-api-smithi072-82692-107-cache"}]': finished 2024-03-26T02:13:49.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:49 smithi173 ceph-mon[33839]: osdmap e538: 8 total, 8 up, 8 in 2024-03-26T02:13:49.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:49 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2374952337' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-107", "overlaypool": "test-rados-api-smithi072-82692-107-cache"}]: dispatch 2024-03-26T02:13:49.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:49 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-107", "overlaypool": "test-rados-api-smithi072-82692-107-cache"}]: dispatch 2024-03-26T02:13:50.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:50 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-107", "overlaypool": "test-rados-api-smithi072-82692-107-cache"}]': finished 2024-03-26T02:13:50.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:50 smithi072 ceph-mon[30177]: osdmap e539: 8 total, 8 up, 8 in 2024-03-26T02:13:50.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:50 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2374952337' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-107-cache", "mode": "writeback"}]: dispatch 2024-03-26T02:13:50.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:50 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-107-cache", "mode": "writeback"}]: dispatch 2024-03-26T02:13:50.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:50 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-107", "overlaypool": "test-rados-api-smithi072-82692-107-cache"}]': finished 2024-03-26T02:13:50.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:50 smithi072 ceph-mon[41619]: osdmap e539: 8 total, 8 up, 8 in 2024-03-26T02:13:50.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:50 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2374952337' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-107-cache", "mode": "writeback"}]: dispatch 2024-03-26T02:13:50.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:50 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-107-cache", "mode": "writeback"}]: dispatch 2024-03-26T02:13:50.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:50 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-107", "overlaypool": "test-rados-api-smithi072-82692-107-cache"}]': finished 2024-03-26T02:13:50.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:50 smithi173 ceph-mon[33839]: osdmap e539: 8 total, 8 up, 8 in 2024-03-26T02:13:50.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:50 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2374952337' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-107-cache", "mode": "writeback"}]: dispatch 2024-03-26T02:13:50.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:50 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-107-cache", "mode": "writeback"}]: dispatch 2024-03-26T02:13:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:51 smithi072 ceph-mon[30177]: pgmap v831: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:13:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:51 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:13:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:51 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-107-cache", "mode": "writeback"}]': finished 2024-03-26T02:13:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:51 smithi072 ceph-mon[30177]: osdmap e540: 8 total, 8 up, 8 in 2024-03-26T02:13:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:51 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2374952337' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-107"}]: dispatch 2024-03-26T02:13:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:51 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-107"}]: dispatch 2024-03-26T02:13:51.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:51 smithi072 ceph-mon[41619]: pgmap v831: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:13:51.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:51 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:13:51.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:51 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-107-cache", "mode": "writeback"}]': finished 2024-03-26T02:13:51.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:51 smithi072 ceph-mon[41619]: osdmap e540: 8 total, 8 up, 8 in 2024-03-26T02:13:51.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:51 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2374952337' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-107"}]: dispatch 2024-03-26T02:13:51.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:51 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-107"}]: dispatch 2024-03-26T02:13:51.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:13:51 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:13:51.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:51 smithi173 ceph-mon[33839]: pgmap v831: 332 pgs: 32 unknown, 300 active+clean; 582 KiB data, 618 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:13:51.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:51 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:13:51.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:51 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-107-cache", "mode": "writeback"}]': finished 2024-03-26T02:13:51.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:51 smithi173 ceph-mon[33839]: osdmap e540: 8 total, 8 up, 8 in 2024-03-26T02:13:51.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:51 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2374952337' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-107"}]: dispatch 2024-03-26T02:13:51.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:51 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-107"}]: dispatch 2024-03-26T02:13:52.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:52 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:13:52.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:52 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-107"}]': finished 2024-03-26T02:13:52.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:52 smithi072 ceph-mon[30177]: osdmap e541: 8 total, 8 up, 8 in 2024-03-26T02:13:52.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:52 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2374952337' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-107", "tierpool": "test-rados-api-smithi072-82692-107-cache"}]: dispatch 2024-03-26T02:13:52.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:52 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-107", "tierpool": "test-rados-api-smithi072-82692-107-cache"}]: dispatch 2024-03-26T02:13:52.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:13:52.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:13:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:52 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:13:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:52 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-107"}]': finished 2024-03-26T02:13:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:52 smithi072 ceph-mon[41619]: osdmap e541: 8 total, 8 up, 8 in 2024-03-26T02:13:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:52 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2374952337' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-107", "tierpool": "test-rados-api-smithi072-82692-107-cache"}]: dispatch 2024-03-26T02:13:52.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:52 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-107", "tierpool": "test-rados-api-smithi072-82692-107-cache"}]: dispatch 2024-03-26T02:13:52.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:13:52.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:13:52.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:52 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:13:52.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:52 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-107"}]': finished 2024-03-26T02:13:52.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:52 smithi173 ceph-mon[33839]: osdmap e541: 8 total, 8 up, 8 in 2024-03-26T02:13:52.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:52 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2374952337' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-107", "tierpool": "test-rados-api-smithi072-82692-107-cache"}]: dispatch 2024-03-26T02:13:52.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:52 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-107", "tierpool": "test-rados-api-smithi072-82692-107-cache"}]: dispatch 2024-03-26T02:13:52.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:13:52.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:13:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:53 smithi072 ceph-mon[30177]: pgmap v834: 332 pgs: 332 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T02:13:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:53 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:13:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:53 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-107", "tierpool": "test-rados-api-smithi072-82692-107-cache"}]': finished 2024-03-26T02:13:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:53 smithi072 ceph-mon[30177]: osdmap e542: 8 total, 8 up, 8 in 2024-03-26T02:13:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:53 smithi072 ceph-mon[41619]: pgmap v834: 332 pgs: 332 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T02:13:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:53 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:13:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:53 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-107", "tierpool": "test-rados-api-smithi072-82692-107-cache"}]': finished 2024-03-26T02:13:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:53 smithi072 ceph-mon[41619]: osdmap e542: 8 total, 8 up, 8 in 2024-03-26T02:13:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:53 smithi173 ceph-mon[33839]: pgmap v834: 332 pgs: 332 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T02:13:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:53 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:13:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:53 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-107", "tierpool": "test-rados-api-smithi072-82692-107-cache"}]': finished 2024-03-26T02:13:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:53 smithi173 ceph-mon[33839]: osdmap e542: 8 total, 8 up, 8 in 2024-03-26T02:13:54.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:54 smithi173 ceph-mon[33839]: osdmap e543: 8 total, 8 up, 8 in 2024-03-26T02:13:54.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:54 smithi173 ceph-mon[33839]: osdmap e544: 8 total, 8 up, 8 in 2024-03-26T02:13:54.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:54 smithi072 ceph-mon[30177]: osdmap e543: 8 total, 8 up, 8 in 2024-03-26T02:13:54.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:54 smithi072 ceph-mon[30177]: osdmap e544: 8 total, 8 up, 8 in 2024-03-26T02:13:54.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:54 smithi072 ceph-mon[41619]: osdmap e543: 8 total, 8 up, 8 in 2024-03-26T02:13:54.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:54 smithi072 ceph-mon[41619]: osdmap e544: 8 total, 8 up, 8 in 2024-03-26T02:13:55.236 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:13:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:13:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:13:55.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:55 smithi173 ceph-mon[33839]: pgmap v837: 300 pgs: 300 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:13:55.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:55 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82692-109"}]: dispatch 2024-03-26T02:13:55.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:55 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82692-109"}]: dispatch 2024-03-26T02:13:55.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:55 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi072-82692-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:13:55.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:55 smithi072 ceph-mon[30177]: pgmap v837: 300 pgs: 300 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:13:55.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:55 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82692-109"}]: dispatch 2024-03-26T02:13:55.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:55 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82692-109"}]: dispatch 2024-03-26T02:13:55.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:55 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi072-82692-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:13:55.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:55 smithi072 ceph-mon[41619]: pgmap v837: 300 pgs: 300 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:13:55.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:55 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82692-109"}]: dispatch 2024-03-26T02:13:55.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:55 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82692-109"}]: dispatch 2024-03-26T02:13:55.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:55 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi072-82692-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:13:56.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:56 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi072-82692-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:13:56.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:56 smithi173 ceph-mon[33839]: osdmap e545: 8 total, 8 up, 8 in 2024-03-26T02:13:56.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:56 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi072-82692-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi072-82692-109"}]: dispatch 2024-03-26T02:13:56.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:56 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi072-82692-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:13:56.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:56 smithi072 ceph-mon[30177]: osdmap e545: 8 total, 8 up, 8 in 2024-03-26T02:13:56.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:56 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi072-82692-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi072-82692-109"}]: dispatch 2024-03-26T02:13:56.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:56 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi072-82692-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:13:56.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:56 smithi072 ceph-mon[41619]: osdmap e545: 8 total, 8 up, 8 in 2024-03-26T02:13:56.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:56 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi072-82692-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi072-82692-109"}]: dispatch 2024-03-26T02:13:57.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:57 smithi173 ceph-mon[33839]: pgmap v840: 268 pgs: 268 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:57.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:57 smithi173 ceph-mon[33839]: osdmap e546: 8 total, 8 up, 8 in 2024-03-26T02:13:57.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:57 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi072-82692-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi072-82692-109"}]': finished 2024-03-26T02:13:57.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:57 smithi173 ceph-mon[33839]: osdmap e547: 8 total, 8 up, 8 in 2024-03-26T02:13:57.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:57 smithi072 ceph-mon[30177]: pgmap v840: 268 pgs: 268 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:57.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:57 smithi072 ceph-mon[30177]: osdmap e546: 8 total, 8 up, 8 in 2024-03-26T02:13:57.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:57 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi072-82692-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi072-82692-109"}]': finished 2024-03-26T02:13:57.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:57 smithi072 ceph-mon[30177]: osdmap e547: 8 total, 8 up, 8 in 2024-03-26T02:13:57.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:57 smithi072 ceph-mon[41619]: pgmap v840: 268 pgs: 268 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:57.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:57 smithi072 ceph-mon[41619]: osdmap e546: 8 total, 8 up, 8 in 2024-03-26T02:13:57.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:57 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi072-82692-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi072-82692-109"}]': finished 2024-03-26T02:13:57.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:57 smithi072 ceph-mon[41619]: osdmap e547: 8 total, 8 up, 8 in 2024-03-26T02:13:59.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:59 smithi173 ceph-mon[33839]: pgmap v843: 276 pgs: 8 unknown, 268 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:59.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:59 smithi173 ceph-mon[33839]: osdmap e548: 8 total, 8 up, 8 in 2024-03-26T02:13:59.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:59 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:59.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:13:59 smithi173 ceph-mon[33839]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:13:59.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:59 smithi072 ceph-mon[41619]: pgmap v843: 276 pgs: 8 unknown, 268 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:59.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:59 smithi072 ceph-mon[41619]: osdmap e548: 8 total, 8 up, 8 in 2024-03-26T02:13:59.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:59 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:59.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:13:59 smithi072 ceph-mon[41619]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:13:59.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:59 smithi072 ceph-mon[30177]: pgmap v843: 276 pgs: 8 unknown, 268 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:13:59.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:59 smithi072 ceph-mon[30177]: osdmap e548: 8 total, 8 up, 8 in 2024-03-26T02:13:59.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:59 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:13:59.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:13:59 smithi072 ceph-mon[30177]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:14:00.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:14:00.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:00 smithi173 ceph-mon[33839]: osdmap e549: 8 total, 8 up, 8 in 2024-03-26T02:14:00.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-109", "tierpool": "test-rados-api-smithi072-82692-109-cache"}]: dispatch 2024-03-26T02:14:00.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-109", "tierpool": "test-rados-api-smithi072-82692-109-cache"}]': finished 2024-03-26T02:14:00.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:00 smithi173 ceph-mon[33839]: osdmap e550: 8 total, 8 up, 8 in 2024-03-26T02:14:00.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-109", "overlaypool": "test-rados-api-smithi072-82692-109-cache"}]: dispatch 2024-03-26T02:14:00.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:14:00.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:00 smithi072 ceph-mon[41619]: osdmap e549: 8 total, 8 up, 8 in 2024-03-26T02:14:00.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-109", "tierpool": "test-rados-api-smithi072-82692-109-cache"}]: dispatch 2024-03-26T02:14:00.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-109", "tierpool": "test-rados-api-smithi072-82692-109-cache"}]': finished 2024-03-26T02:14:00.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:00 smithi072 ceph-mon[41619]: osdmap e550: 8 total, 8 up, 8 in 2024-03-26T02:14:00.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-109", "overlaypool": "test-rados-api-smithi072-82692-109-cache"}]: dispatch 2024-03-26T02:14:00.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:14:00.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:00 smithi072 ceph-mon[30177]: osdmap e549: 8 total, 8 up, 8 in 2024-03-26T02:14:00.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-109", "tierpool": "test-rados-api-smithi072-82692-109-cache"}]: dispatch 2024-03-26T02:14:00.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-109", "tierpool": "test-rados-api-smithi072-82692-109-cache"}]': finished 2024-03-26T02:14:00.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:00 smithi072 ceph-mon[30177]: osdmap e550: 8 total, 8 up, 8 in 2024-03-26T02:14:00.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-109", "overlaypool": "test-rados-api-smithi072-82692-109-cache"}]: dispatch 2024-03-26T02:14:01.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:14:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:14:01.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:01 smithi173 ceph-mon[33839]: pgmap v846: 308 pgs: 40 unknown, 268 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:14:01.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-109", "overlaypool": "test-rados-api-smithi072-82692-109-cache"}]': finished 2024-03-26T02:14:01.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:01 smithi173 ceph-mon[33839]: osdmap e551: 8 total, 8 up, 8 in 2024-03-26T02:14:01.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-109-cache", "mode": "writeback"}]: dispatch 2024-03-26T02:14:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:01 smithi072 ceph-mon[30177]: pgmap v846: 308 pgs: 40 unknown, 268 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:14:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-109", "overlaypool": "test-rados-api-smithi072-82692-109-cache"}]': finished 2024-03-26T02:14:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:01 smithi072 ceph-mon[30177]: osdmap e551: 8 total, 8 up, 8 in 2024-03-26T02:14:01.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-109-cache", "mode": "writeback"}]: dispatch 2024-03-26T02:14:01.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:01 smithi072 ceph-mon[41619]: pgmap v846: 308 pgs: 40 unknown, 268 active+clean; 582 KiB data, 619 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:14:01.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-109", "overlaypool": "test-rados-api-smithi072-82692-109-cache"}]': finished 2024-03-26T02:14:01.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:01 smithi072 ceph-mon[41619]: osdmap e551: 8 total, 8 up, 8 in 2024-03-26T02:14:01.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-109-cache", "mode": "writeback"}]: dispatch 2024-03-26T02:14:02.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:02 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:14:02.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:02 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:14:02.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-109-cache", "mode": "writeback"}]': finished 2024-03-26T02:14:02.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:02 smithi173 ceph-mon[33839]: osdmap e552: 8 total, 8 up, 8 in 2024-03-26T02:14:02.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:14:02.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:02 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:14:02.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:02 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:14:02.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-109-cache", "mode": "writeback"}]': finished 2024-03-26T02:14:02.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:02 smithi072 ceph-mon[30177]: osdmap e552: 8 total, 8 up, 8 in 2024-03-26T02:14:02.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:14:02.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:02 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:14:02.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:02 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:14:02.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-109-cache", "mode": "writeback"}]': finished 2024-03-26T02:14:02.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:02 smithi072 ceph-mon[41619]: osdmap e552: 8 total, 8 up, 8 in 2024-03-26T02:14:02.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:14:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:03 smithi173 ceph-mon[33839]: pgmap v849: 308 pgs: 308 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:14:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-26T02:14:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:03 smithi173 ceph-mon[33839]: osdmap e553: 8 total, 8 up, 8 in 2024-03-26T02:14:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:14:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:03 smithi072 ceph-mon[30177]: pgmap v849: 308 pgs: 308 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:14:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-26T02:14:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:03 smithi072 ceph-mon[30177]: osdmap e553: 8 total, 8 up, 8 in 2024-03-26T02:14:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:14:03.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:03 smithi072 ceph-mon[41619]: pgmap v849: 308 pgs: 308 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:14:03.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-26T02:14:03.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:03 smithi072 ceph-mon[41619]: osdmap e553: 8 total, 8 up, 8 in 2024-03-26T02:14:03.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:14:05.257 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:14:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:14:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:14:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:05 smithi173 ceph-mon[33839]: pgmap v852: 308 pgs: 308 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:14:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:14:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:05 smithi173 ceph-mon[33839]: osdmap e554: 8 total, 8 up, 8 in 2024-03-26T02:14:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T02:14:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:05 smithi072 ceph-mon[30177]: pgmap v852: 308 pgs: 308 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:14:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:14:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:05 smithi072 ceph-mon[30177]: osdmap e554: 8 total, 8 up, 8 in 2024-03-26T02:14:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T02:14:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:05 smithi072 ceph-mon[41619]: pgmap v852: 308 pgs: 308 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:14:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:14:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:05 smithi072 ceph-mon[41619]: osdmap e554: 8 total, 8 up, 8 in 2024-03-26T02:14:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T02:14:06.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:06 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:14:06.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T02:14:06.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:06 smithi173 ceph-mon[33839]: osdmap e555: 8 total, 8 up, 8 in 2024-03-26T02:14:06.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-26T02:14:06.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-26T02:14:06.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:06 smithi173 ceph-mon[33839]: osdmap e556: 8 total, 8 up, 8 in 2024-03-26T02:14:06.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:06 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:14:06.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T02:14:06.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:06 smithi072 ceph-mon[30177]: osdmap e555: 8 total, 8 up, 8 in 2024-03-26T02:14:06.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-26T02:14:06.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-26T02:14:06.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:06 smithi072 ceph-mon[30177]: osdmap e556: 8 total, 8 up, 8 in 2024-03-26T02:14:06.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:06 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:14:06.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T02:14:06.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:06 smithi072 ceph-mon[41619]: osdmap e555: 8 total, 8 up, 8 in 2024-03-26T02:14:06.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-26T02:14:06.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-26T02:14:06.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:06 smithi072 ceph-mon[41619]: osdmap e556: 8 total, 8 up, 8 in 2024-03-26T02:14:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:07 smithi173 ceph-mon[33839]: pgmap v855: 308 pgs: 308 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:14:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-109"}]: dispatch 2024-03-26T02:14:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:07 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:14:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:07 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:14:07.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:07 smithi072 ceph-mon[30177]: pgmap v855: 308 pgs: 308 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:14:07.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-109"}]: dispatch 2024-03-26T02:14:07.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:07 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:14:07.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:07 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:14:07.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:07 smithi072 ceph-mon[41619]: pgmap v855: 308 pgs: 308 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:14:07.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-109"}]: dispatch 2024-03-26T02:14:07.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:07 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:14:07.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:07 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:14:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-109"}]': finished 2024-03-26T02:14:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:08 smithi173 ceph-mon[33839]: osdmap e557: 8 total, 8 up, 8 in 2024-03-26T02:14:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-109", "tierpool": "test-rados-api-smithi072-82692-109-cache"}]: dispatch 2024-03-26T02:14:08.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-109", "tierpool": "test-rados-api-smithi072-82692-109-cache"}]': finished 2024-03-26T02:14:08.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:08 smithi173 ceph-mon[33839]: osdmap e558: 8 total, 8 up, 8 in 2024-03-26T02:14:08.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-109"}]': finished 2024-03-26T02:14:08.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:08 smithi072 ceph-mon[30177]: osdmap e557: 8 total, 8 up, 8 in 2024-03-26T02:14:08.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-109", "tierpool": "test-rados-api-smithi072-82692-109-cache"}]: dispatch 2024-03-26T02:14:08.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-109", "tierpool": "test-rados-api-smithi072-82692-109-cache"}]': finished 2024-03-26T02:14:08.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:08 smithi072 ceph-mon[30177]: osdmap e558: 8 total, 8 up, 8 in 2024-03-26T02:14:08.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-109"}]': finished 2024-03-26T02:14:08.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:08 smithi072 ceph-mon[41619]: osdmap e557: 8 total, 8 up, 8 in 2024-03-26T02:14:08.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-109", "tierpool": "test-rados-api-smithi072-82692-109-cache"}]: dispatch 2024-03-26T02:14:08.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-109", "tierpool": "test-rados-api-smithi072-82692-109-cache"}]': finished 2024-03-26T02:14:08.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:08 smithi072 ceph-mon[41619]: osdmap e558: 8 total, 8 up, 8 in 2024-03-26T02:14:09.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:09 smithi173 ceph-mon[33839]: pgmap v858: 308 pgs: 308 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:14:09.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:09 smithi173 ceph-mon[33839]: osdmap e559: 8 total, 8 up, 8 in 2024-03-26T02:14:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:09 smithi072 ceph-mon[30177]: pgmap v858: 308 pgs: 308 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:14:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:09 smithi072 ceph-mon[30177]: osdmap e559: 8 total, 8 up, 8 in 2024-03-26T02:14:09.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:09 smithi072 ceph-mon[41619]: pgmap v858: 308 pgs: 308 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:14:09.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:09 smithi072 ceph-mon[41619]: osdmap e559: 8 total, 8 up, 8 in 2024-03-26T02:14:11.365 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:14:11 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:14:11.365 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:11 smithi173 ceph-mon[33839]: pgmap v861: 276 pgs: 276 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:14:11.365 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:11 smithi173 ceph-mon[33839]: osdmap e560: 8 total, 8 up, 8 in 2024-03-26T02:14:11.366 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:11 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82692-109"}]: dispatch 2024-03-26T02:14:11.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:11 smithi072 ceph-mon[30177]: pgmap v861: 276 pgs: 276 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:14:11.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:11 smithi072 ceph-mon[30177]: osdmap e560: 8 total, 8 up, 8 in 2024-03-26T02:14:11.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:11 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82692-109"}]: dispatch 2024-03-26T02:14:11.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:11 smithi072 ceph-mon[41619]: pgmap v861: 276 pgs: 276 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:14:11.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:11 smithi072 ceph-mon[41619]: osdmap e560: 8 total, 8 up, 8 in 2024-03-26T02:14:11.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:11 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82692-109"}]: dispatch 2024-03-26T02:14:12.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:12 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:14:12.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82692-109"}]': finished 2024-03-26T02:14:12.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:12 smithi173 ceph-mon[33839]: osdmap e561: 8 total, 8 up, 8 in 2024-03-26T02:14:12.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82692-109"}]: dispatch 2024-03-26T02:14:12.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82692-109"}]': finished 2024-03-26T02:14:12.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:12 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:14:12.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82692-109"}]': finished 2024-03-26T02:14:12.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:12 smithi072 ceph-mon[41619]: osdmap e561: 8 total, 8 up, 8 in 2024-03-26T02:14:12.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82692-109"}]: dispatch 2024-03-26T02:14:12.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82692-109"}]': finished 2024-03-26T02:14:12.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:12 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:14:12.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82692-109"}]': finished 2024-03-26T02:14:12.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:12 smithi072 ceph-mon[30177]: osdmap e561: 8 total, 8 up, 8 in 2024-03-26T02:14:12.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82692-109"}]: dispatch 2024-03-26T02:14:12.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2854330857' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82692-109"}]': finished 2024-03-26T02:14:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:13 smithi173 ceph-mon[33839]: pgmap v864: 268 pgs: 268 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:14:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:13 smithi173 ceph-mon[33839]: osdmap e562: 8 total, 8 up, 8 in 2024-03-26T02:14:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:13 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:14:13.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:13 smithi072 ceph-mon[30177]: pgmap v864: 268 pgs: 268 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:14:13.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:13 smithi072 ceph-mon[30177]: osdmap e562: 8 total, 8 up, 8 in 2024-03-26T02:14:13.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:13 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:14:13.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:13 smithi072 ceph-mon[41619]: pgmap v864: 268 pgs: 268 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:14:13.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:13 smithi072 ceph-mon[41619]: osdmap e562: 8 total, 8 up, 8 in 2024-03-26T02:14:13.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:13 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:14:14.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:14 smithi173 ceph-mon[33839]: osdmap e563: 8 total, 8 up, 8 in 2024-03-26T02:14:14.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:14 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi072-82692-104"}]: dispatch 2024-03-26T02:14:14.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:14 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi072-82692-104"}]': finished 2024-03-26T02:14:14.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:14 smithi173 ceph-mon[33839]: osdmap e564: 8 total, 8 up, 8 in 2024-03-26T02:14:14.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:14 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi072-82692-104"}]: dispatch 2024-03-26T02:14:14.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:14 smithi072 ceph-mon[30177]: osdmap e563: 8 total, 8 up, 8 in 2024-03-26T02:14:14.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:14 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi072-82692-104"}]: dispatch 2024-03-26T02:14:14.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:14 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi072-82692-104"}]': finished 2024-03-26T02:14:14.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:14 smithi072 ceph-mon[30177]: osdmap e564: 8 total, 8 up, 8 in 2024-03-26T02:14:14.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:14 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi072-82692-104"}]: dispatch 2024-03-26T02:14:14.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:14 smithi072 ceph-mon[41619]: osdmap e563: 8 total, 8 up, 8 in 2024-03-26T02:14:14.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:14 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi072-82692-104"}]: dispatch 2024-03-26T02:14:14.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:14 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi072-82692-104"}]': finished 2024-03-26T02:14:14.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:14 smithi072 ceph-mon[41619]: osdmap e564: 8 total, 8 up, 8 in 2024-03-26T02:14:14.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:14 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi072-82692-104"}]: dispatch 2024-03-26T02:14:15.310 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:14:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:14:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:14:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:15 smithi173 ceph-mon[33839]: pgmap v867: 260 pgs: 260 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T02:14:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi072-82692-104"}]': finished 2024-03-26T02:14:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:15 smithi173 ceph-mon[33839]: osdmap e565: 8 total, 8 up, 8 in 2024-03-26T02:14:15.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:15 smithi072 ceph-mon[30177]: pgmap v867: 260 pgs: 260 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T02:14:15.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi072-82692-104"}]': finished 2024-03-26T02:14:15.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:15 smithi072 ceph-mon[30177]: osdmap e565: 8 total, 8 up, 8 in 2024-03-26T02:14:15.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:15 smithi072 ceph-mon[41619]: pgmap v867: 260 pgs: 260 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T02:14:15.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1161296078' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi072-82692-104"}]': finished 2024-03-26T02:14:15.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:15 smithi072 ceph-mon[41619]: osdmap e565: 8 total, 8 up, 8 in 2024-03-26T02:14:16.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:16 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:14:16.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:16 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:14:16.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:16 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi072-82692-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:14:16.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:16 smithi072 ceph-mon[30177]: pgmap v870: 260 pgs: 260 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T02:14:16.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:16 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:14:16.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:16 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:14:16.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:16 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi072-82692-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:14:16.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:16 smithi072 ceph-mon[41619]: pgmap v870: 260 pgs: 260 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T02:14:16.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:16 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:14:16.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:16 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:14:16.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:16 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi072-82692-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T02:14:16.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:16 smithi173 ceph-mon[33839]: pgmap v870: 260 pgs: 260 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T02:14:17.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:17 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi072-82692-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:14:17.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:17 smithi072 ceph-mon[30177]: osdmap e566: 8 total, 8 up, 8 in 2024-03-26T02:14:17.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:17 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi072-82692-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:14:17.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:17 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi072-82692-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:14:17.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:17 smithi072 ceph-mon[41619]: osdmap e566: 8 total, 8 up, 8 in 2024-03-26T02:14:17.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:17 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi072-82692-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:14:17.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:17 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi072-82692-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T02:14:17.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:17 smithi173 ceph-mon[33839]: osdmap e566: 8 total, 8 up, 8 in 2024-03-26T02:14:17.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:17 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi072-82692-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:14:18.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:18 smithi072 ceph-mon[30177]: osdmap e567: 8 total, 8 up, 8 in 2024-03-26T02:14:18.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:18 smithi072 ceph-mon[30177]: pgmap v873: 260 pgs: 260 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:14:18.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:18 smithi072 ceph-mon[41619]: osdmap e567: 8 total, 8 up, 8 in 2024-03-26T02:14:18.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:18 smithi072 ceph-mon[41619]: pgmap v873: 260 pgs: 260 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:14:18.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:18 smithi173 ceph-mon[33839]: osdmap e567: 8 total, 8 up, 8 in 2024-03-26T02:14:18.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:18 smithi173 ceph-mon[33839]: pgmap v873: 260 pgs: 260 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:14:19.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi072-82692-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:14:19.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:19 smithi072 ceph-mon[30177]: osdmap e568: 8 total, 8 up, 8 in 2024-03-26T02:14:19.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi072-82692-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:14:19.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:19 smithi072 ceph-mon[41619]: osdmap e568: 8 total, 8 up, 8 in 2024-03-26T02:14:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi072-82692-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:14:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:19 smithi173 ceph-mon[33839]: osdmap e568: 8 total, 8 up, 8 in 2024-03-26T02:14:20.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:20 smithi072 ceph-mon[30177]: osdmap e569: 8 total, 8 up, 8 in 2024-03-26T02:14:20.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:14:20.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:20 smithi072 ceph-mon[30177]: pgmap v876: 300 pgs: 40 unknown, 260 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:14:20.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:20 smithi072 ceph-mon[41619]: osdmap e569: 8 total, 8 up, 8 in 2024-03-26T02:14:20.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:14:20.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:20 smithi072 ceph-mon[41619]: pgmap v876: 300 pgs: 40 unknown, 260 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:14:20.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:20 smithi173 ceph-mon[33839]: osdmap e569: 8 total, 8 up, 8 in 2024-03-26T02:14:20.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:14:20.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:20 smithi173 ceph-mon[33839]: pgmap v876: 300 pgs: 40 unknown, 260 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:14:21.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:14:21 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:14:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:14:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:21 smithi173 ceph-mon[33839]: osdmap e570: 8 total, 8 up, 8 in 2024-03-26T02:14:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:21 smithi173 ceph-mon[33839]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:14:21.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:14:21.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:21 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:14:21.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:14:21.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:21 smithi173 ceph-mon[33839]: osdmap e571: 8 total, 8 up, 8 in 2024-03-26T02:14:21.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-112"}]: dispatch 2024-03-26T02:14:21.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:14:21.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:21 smithi072 ceph-mon[30177]: osdmap e570: 8 total, 8 up, 8 in 2024-03-26T02:14:21.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:21 smithi072 ceph-mon[30177]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:14:21.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:14:21.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:21 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:14:21.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:14:21.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:21 smithi072 ceph-mon[30177]: osdmap e571: 8 total, 8 up, 8 in 2024-03-26T02:14:21.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-112"}]: dispatch 2024-03-26T02:14:21.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:14:21.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:21 smithi072 ceph-mon[41619]: osdmap e570: 8 total, 8 up, 8 in 2024-03-26T02:14:21.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:21 smithi072 ceph-mon[41619]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:14:21.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:14:21.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:21 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:14:21.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:14:21.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:21 smithi072 ceph-mon[41619]: osdmap e571: 8 total, 8 up, 8 in 2024-03-26T02:14:21.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-112"}]: dispatch 2024-03-26T02:14:22.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:22 smithi072 ceph-mon[30177]: pgmap v879: 300 pgs: 300 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T02:14:22.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:22 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:14:22.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:22 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-112"}]': finished 2024-03-26T02:14:22.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:22 smithi072 ceph-mon[30177]: osdmap e572: 8 total, 8 up, 8 in 2024-03-26T02:14:22.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:22 smithi072 ceph-mon[41619]: pgmap v879: 300 pgs: 300 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T02:14:22.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:22 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:14:22.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:22 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-112"}]': finished 2024-03-26T02:14:22.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:22 smithi072 ceph-mon[41619]: osdmap e572: 8 total, 8 up, 8 in 2024-03-26T02:14:22.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:22 smithi173 ceph-mon[33839]: pgmap v879: 300 pgs: 300 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T02:14:22.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:22 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:14:22.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:22 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-112"}]': finished 2024-03-26T02:14:22.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:22 smithi173 ceph-mon[33839]: osdmap e572: 8 total, 8 up, 8 in 2024-03-26T02:14:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:23 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:14:23.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:23 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:14:23.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:23 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:14:24.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:24 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:14:24.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:24 smithi072 ceph-mon[41619]: osdmap e573: 8 total, 8 up, 8 in 2024-03-26T02:14:24.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:24 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-112"}]: dispatch 2024-03-26T02:14:24.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:24 smithi072 ceph-mon[41619]: pgmap v882: 300 pgs: 300 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T02:14:24.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:24 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:14:24.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:24 smithi072 ceph-mon[30177]: osdmap e573: 8 total, 8 up, 8 in 2024-03-26T02:14:24.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:24 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-112"}]: dispatch 2024-03-26T02:14:24.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:24 smithi072 ceph-mon[30177]: pgmap v882: 300 pgs: 300 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T02:14:24.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:24 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:14:24.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:24 smithi173 ceph-mon[33839]: osdmap e573: 8 total, 8 up, 8 in 2024-03-26T02:14:24.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:24 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-112"}]: dispatch 2024-03-26T02:14:24.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:24 smithi173 ceph-mon[33839]: pgmap v882: 300 pgs: 300 active+clean; 582 KiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T02:14:25.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:14:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:14:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:14:25.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-112"}]': finished 2024-03-26T02:14:25.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:25 smithi072 ceph-mon[30177]: osdmap e574: 8 total, 8 up, 8 in 2024-03-26T02:14:25.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-112"}]': finished 2024-03-26T02:14:25.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:25 smithi072 ceph-mon[41619]: osdmap e574: 8 total, 8 up, 8 in 2024-03-26T02:14:26.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-112"}]': finished 2024-03-26T02:14:26.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:25 smithi173 ceph-mon[33839]: osdmap e574: 8 total, 8 up, 8 in 2024-03-26T02:14:26.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:26 smithi072 ceph-mon[30177]: osdmap e575: 8 total, 8 up, 8 in 2024-03-26T02:14:26.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:26 smithi072 ceph-mon[30177]: pgmap v885: 268 pgs: 268 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T02:14:26.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:26 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:14:26.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:26 smithi072 ceph-mon[30177]: osdmap e576: 8 total, 8 up, 8 in 2024-03-26T02:14:26.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:26 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:14:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:26 smithi072 ceph-mon[41619]: osdmap e575: 8 total, 8 up, 8 in 2024-03-26T02:14:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:26 smithi072 ceph-mon[41619]: pgmap v885: 268 pgs: 268 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T02:14:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:26 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:14:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:26 smithi072 ceph-mon[41619]: osdmap e576: 8 total, 8 up, 8 in 2024-03-26T02:14:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:26 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:14:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:26 smithi173 ceph-mon[33839]: osdmap e575: 8 total, 8 up, 8 in 2024-03-26T02:14:27.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:26 smithi173 ceph-mon[33839]: pgmap v885: 268 pgs: 268 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T02:14:27.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:26 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:14:27.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:26 smithi173 ceph-mon[33839]: osdmap e576: 8 total, 8 up, 8 in 2024-03-26T02:14:27.065 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:26 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:14:28.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:28 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:14:28.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:28 smithi072 ceph-mon[30177]: osdmap e577: 8 total, 8 up, 8 in 2024-03-26T02:14:28.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:28 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:14:28.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:28 smithi072 ceph-mon[30177]: pgmap v888: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T02:14:28.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:28 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:14:28.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:28 smithi072 ceph-mon[41619]: osdmap e577: 8 total, 8 up, 8 in 2024-03-26T02:14:28.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:28 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:14:28.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:28 smithi072 ceph-mon[41619]: pgmap v888: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T02:14:28.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:28 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:14:28.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:28 smithi173 ceph-mon[33839]: osdmap e577: 8 total, 8 up, 8 in 2024-03-26T02:14:28.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:28 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:14:28.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:28 smithi173 ceph-mon[33839]: pgmap v888: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T02:14:29.723 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:29 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:14:29.723 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:29 smithi072 ceph-mon[30177]: osdmap e578: 8 total, 8 up, 8 in 2024-03-26T02:14:29.723 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:29 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-114"}]: dispatch 2024-03-26T02:14:29.723 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:29 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:14:29.724 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:29 smithi072 ceph-mon[41619]: osdmap e578: 8 total, 8 up, 8 in 2024-03-26T02:14:29.724 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:29 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-114"}]: dispatch 2024-03-26T02:14:29.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:29 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:14:29.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:29 smithi173 ceph-mon[33839]: osdmap e578: 8 total, 8 up, 8 in 2024-03-26T02:14:29.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:29 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-114"}]: dispatch 2024-03-26T02:14:30.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-114"}]': finished 2024-03-26T02:14:30.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:30 smithi072 ceph-mon[30177]: osdmap e579: 8 total, 8 up, 8 in 2024-03-26T02:14:30.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-114", "mode": "writeback"}]: dispatch 2024-03-26T02:14:30.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:30 smithi072 ceph-mon[30177]: pgmap v891: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:14:30.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:30 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:14:30.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-114", "mode": "writeback"}]': finished 2024-03-26T02:14:30.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:30 smithi072 ceph-mon[30177]: osdmap e580: 8 total, 8 up, 8 in 2024-03-26T02:14:30.722 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-114"}]': finished 2024-03-26T02:14:30.723 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:30 smithi072 ceph-mon[41619]: osdmap e579: 8 total, 8 up, 8 in 2024-03-26T02:14:30.723 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-114", "mode": "writeback"}]: dispatch 2024-03-26T02:14:30.723 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:30 smithi072 ceph-mon[41619]: pgmap v891: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:14:30.723 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:30 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:14:30.723 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-114", "mode": "writeback"}]': finished 2024-03-26T02:14:30.723 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:30 smithi072 ceph-mon[41619]: osdmap e580: 8 total, 8 up, 8 in 2024-03-26T02:14:30.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-114"}]': finished 2024-03-26T02:14:30.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:30 smithi173 ceph-mon[33839]: osdmap e579: 8 total, 8 up, 8 in 2024-03-26T02:14:30.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-114", "mode": "writeback"}]: dispatch 2024-03-26T02:14:30.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:30 smithi173 ceph-mon[33839]: pgmap v891: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:14:30.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:30 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:14:30.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-114", "mode": "writeback"}]': finished 2024-03-26T02:14:30.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:30 smithi173 ceph-mon[33839]: osdmap e580: 8 total, 8 up, 8 in 2024-03-26T02:14:31.426 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:14:31 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:14:31.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:31 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:14:31.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:31 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:14:31.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:31 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:14:31.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:31 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:14:31.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:31 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:14:31.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:31 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:14:32.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:14:32.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:32 smithi072 ceph-mon[30177]: osdmap e581: 8 total, 8 up, 8 in 2024-03-26T02:14:32.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-114"}]: dispatch 2024-03-26T02:14:32.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:32 smithi072 ceph-mon[30177]: pgmap v894: 300 pgs: 300 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T02:14:32.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:14:32.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:32 smithi072 ceph-mon[41619]: osdmap e581: 8 total, 8 up, 8 in 2024-03-26T02:14:32.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-114"}]: dispatch 2024-03-26T02:14:32.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:32 smithi072 ceph-mon[41619]: pgmap v894: 300 pgs: 300 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T02:14:32.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:14:32.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:32 smithi173 ceph-mon[33839]: osdmap e581: 8 total, 8 up, 8 in 2024-03-26T02:14:32.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-114"}]: dispatch 2024-03-26T02:14:32.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:32 smithi173 ceph-mon[33839]: pgmap v894: 300 pgs: 300 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T02:14:33.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:33 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:14:33.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:33 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-114"}]': finished 2024-03-26T02:14:33.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:33 smithi072 ceph-mon[30177]: osdmap e582: 8 total, 8 up, 8 in 2024-03-26T02:14:33.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:33 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:14:33.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:33 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-114"}]': finished 2024-03-26T02:14:33.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:33 smithi072 ceph-mon[41619]: osdmap e582: 8 total, 8 up, 8 in 2024-03-26T02:14:34.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:33 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:14:34.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:33 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-114"}]': finished 2024-03-26T02:14:34.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:33 smithi173 ceph-mon[33839]: osdmap e582: 8 total, 8 up, 8 in 2024-03-26T02:14:34.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:34 smithi072 ceph-mon[30177]: osdmap e583: 8 total, 8 up, 8 in 2024-03-26T02:14:34.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:34 smithi072 ceph-mon[30177]: pgmap v897: 268 pgs: 268 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:14:34.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:34 smithi072 ceph-mon[30177]: osdmap e584: 8 total, 8 up, 8 in 2024-03-26T02:14:34.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:34 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:14:34.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:34 smithi072 ceph-mon[41619]: osdmap e583: 8 total, 8 up, 8 in 2024-03-26T02:14:34.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:34 smithi072 ceph-mon[41619]: pgmap v897: 268 pgs: 268 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:14:34.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:34 smithi072 ceph-mon[41619]: osdmap e584: 8 total, 8 up, 8 in 2024-03-26T02:14:34.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:34 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:14:35.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:34 smithi173 ceph-mon[33839]: osdmap e583: 8 total, 8 up, 8 in 2024-03-26T02:14:35.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:34 smithi173 ceph-mon[33839]: pgmap v897: 268 pgs: 268 active+clean; 582 KiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:14:35.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:34 smithi173 ceph-mon[33839]: osdmap e584: 8 total, 8 up, 8 in 2024-03-26T02:14:35.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:34 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:14:35.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:14:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:14:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:14:36.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:36 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:14:36.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:36 smithi072 ceph-mon[30177]: osdmap e585: 8 total, 8 up, 8 in 2024-03-26T02:14:36.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:36 smithi072 ceph-mon[30177]: pgmap v900: 300 pgs: 300 active+clean; 582 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1023 B/s wr, 1 op/s 2024-03-26T02:14:36.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:36 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:14:36.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:36 smithi072 ceph-mon[41619]: osdmap e585: 8 total, 8 up, 8 in 2024-03-26T02:14:36.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:36 smithi072 ceph-mon[41619]: pgmap v900: 300 pgs: 300 active+clean; 582 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1023 B/s wr, 1 op/s 2024-03-26T02:14:36.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:36 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:14:36.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:36 smithi173 ceph-mon[33839]: osdmap e585: 8 total, 8 up, 8 in 2024-03-26T02:14:36.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:36 smithi173 ceph-mon[33839]: pgmap v900: 300 pgs: 300 active+clean; 582 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1023 B/s wr, 1 op/s 2024-03-26T02:14:37.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:37 smithi072 ceph-mon[30177]: osdmap e586: 8 total, 8 up, 8 in 2024-03-26T02:14:37.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:37 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:14:37.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:37 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:14:37.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:37 smithi072 ceph-mon[41619]: osdmap e586: 8 total, 8 up, 8 in 2024-03-26T02:14:37.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:37 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:14:37.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:37 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:14:37.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:37 smithi173 ceph-mon[33839]: osdmap e586: 8 total, 8 up, 8 in 2024-03-26T02:14:37.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:37 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:14:37.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:37 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:14:38.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:38 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:14:38.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:38 smithi173 ceph-mon[33839]: osdmap e587: 8 total, 8 up, 8 in 2024-03-26T02:14:38.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:38 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-116"}]: dispatch 2024-03-26T02:14:38.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:38 smithi173 ceph-mon[33839]: pgmap v903: 300 pgs: 300 active+clean; 582 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T02:14:38.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:38 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-116"}]': finished 2024-03-26T02:14:38.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:38 smithi173 ceph-mon[33839]: osdmap e588: 8 total, 8 up, 8 in 2024-03-26T02:14:38.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:38 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-116", "mode": "writeback"}]: dispatch 2024-03-26T02:14:38.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:38 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:14:38.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:38 smithi072 ceph-mon[30177]: osdmap e587: 8 total, 8 up, 8 in 2024-03-26T02:14:38.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:38 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-116"}]: dispatch 2024-03-26T02:14:38.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:38 smithi072 ceph-mon[30177]: pgmap v903: 300 pgs: 300 active+clean; 582 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T02:14:38.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:38 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-116"}]': finished 2024-03-26T02:14:38.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:38 smithi072 ceph-mon[30177]: osdmap e588: 8 total, 8 up, 8 in 2024-03-26T02:14:38.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:38 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-116", "mode": "writeback"}]: dispatch 2024-03-26T02:14:38.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:38 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:14:38.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:38 smithi072 ceph-mon[41619]: osdmap e587: 8 total, 8 up, 8 in 2024-03-26T02:14:38.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:38 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-116"}]: dispatch 2024-03-26T02:14:38.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:38 smithi072 ceph-mon[41619]: pgmap v903: 300 pgs: 300 active+clean; 582 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T02:14:38.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:38 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-116"}]': finished 2024-03-26T02:14:38.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:38 smithi072 ceph-mon[41619]: osdmap e588: 8 total, 8 up, 8 in 2024-03-26T02:14:38.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:38 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-116", "mode": "writeback"}]: dispatch 2024-03-26T02:14:39.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:39 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:14:39.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-116", "mode": "writeback"}]': finished 2024-03-26T02:14:39.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:39 smithi173 ceph-mon[33839]: osdmap e589: 8 total, 8 up, 8 in 2024-03-26T02:14:39.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:39 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:14:39.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-116", "mode": "writeback"}]': finished 2024-03-26T02:14:39.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:39 smithi072 ceph-mon[41619]: osdmap e589: 8 total, 8 up, 8 in 2024-03-26T02:14:39.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:39 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:14:39.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-116", "mode": "writeback"}]': finished 2024-03-26T02:14:39.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:39 smithi072 ceph-mon[30177]: osdmap e589: 8 total, 8 up, 8 in 2024-03-26T02:14:40.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "315.6"}]: dispatch 2024-03-26T02:14:40.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:40 smithi173 ceph-mon[33839]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "315.6"}]: dispatch 2024-03-26T02:14:40.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:40 smithi173 ceph-mon[33839]: 315.6 scrub starts 2024-03-26T02:14:40.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:40 smithi173 ceph-mon[33839]: 315.6 scrub ok 2024-03-26T02:14:40.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:40 smithi173 ceph-mon[33839]: pgmap v906: 300 pgs: 300 active+clean; 582 KiB data, 596 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:14:40.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "315.6"}]: dispatch 2024-03-26T02:14:40.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:40 smithi072 ceph-mon[30177]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "315.6"}]: dispatch 2024-03-26T02:14:40.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:40 smithi072 ceph-mon[30177]: 315.6 scrub starts 2024-03-26T02:14:40.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:40 smithi072 ceph-mon[30177]: 315.6 scrub ok 2024-03-26T02:14:40.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:40 smithi072 ceph-mon[30177]: pgmap v906: 300 pgs: 300 active+clean; 582 KiB data, 596 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:14:40.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "315.6"}]: dispatch 2024-03-26T02:14:40.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:40 smithi072 ceph-mon[41619]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "315.6"}]: dispatch 2024-03-26T02:14:40.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:40 smithi072 ceph-mon[41619]: 315.6 scrub starts 2024-03-26T02:14:40.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:40 smithi072 ceph-mon[41619]: 315.6 scrub ok 2024-03-26T02:14:40.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:40 smithi072 ceph-mon[41619]: pgmap v906: 300 pgs: 300 active+clean; 582 KiB data, 596 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:14:41.481 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:14:41 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:14:41.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:41 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:14:41.870 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:41 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:14:41.871 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:41 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:14:42.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:42 smithi173 ceph-mon[33839]: pgmap v907: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 585 B/s wr, 2 op/s 2024-03-26T02:14:42.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:42 smithi072 ceph-mon[30177]: pgmap v907: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 585 B/s wr, 2 op/s 2024-03-26T02:14:42.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:42 smithi072 ceph-mon[41619]: pgmap v907: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 585 B/s wr, 2 op/s 2024-03-26T02:14:45.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:44 smithi173 ceph-mon[33839]: pgmap v908: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 492 B/s wr, 2 op/s 2024-03-26T02:14:45.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:44 smithi072 ceph-mon[30177]: pgmap v908: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 492 B/s wr, 2 op/s 2024-03-26T02:14:45.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:44 smithi072 ceph-mon[41619]: pgmap v908: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 492 B/s wr, 2 op/s 2024-03-26T02:14:45.220 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:14:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:14:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:14:47.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:46 smithi173 ceph-mon[33839]: pgmap v909: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 383 B/s wr, 2 op/s 2024-03-26T02:14:47.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:46 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:14:47.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:46 smithi072 ceph-mon[30177]: pgmap v909: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 383 B/s wr, 2 op/s 2024-03-26T02:14:47.124 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:46 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:14:47.124 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:46 smithi072 ceph-mon[41619]: pgmap v909: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 383 B/s wr, 2 op/s 2024-03-26T02:14:47.124 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:46 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:14:49.021 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:48 smithi173 ceph-mon[33839]: pgmap v910: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 332 B/s wr, 1 op/s 2024-03-26T02:14:49.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:48 smithi072 ceph-mon[30177]: pgmap v910: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 332 B/s wr, 1 op/s 2024-03-26T02:14:49.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:48 smithi072 ceph-mon[41619]: pgmap v910: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 332 B/s wr, 1 op/s 2024-03-26T02:14:50.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:14:50.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:14:50.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.2", "id": [2, 0]}]: dispatch 2024-03-26T02:14:50.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.7", "id": [3, 1]}]: dispatch 2024-03-26T02:14:50.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.b", "id": [2, 1]}]: dispatch 2024-03-26T02:14:50.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.c", "id": [7, 1]}]: dispatch 2024-03-26T02:14:50.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.1b", "id": [7, 4]}]: dispatch 2024-03-26T02:14:50.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.1d", "id": [2, 0]}]: dispatch 2024-03-26T02:14:50.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:50 smithi173 ceph-mon[33839]: pgmap v911: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 299 B/s wr, 1 op/s 2024-03-26T02:14:50.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:14:50.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:14:50.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.2", "id": [2, 0]}]: dispatch 2024-03-26T02:14:50.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.7", "id": [3, 1]}]: dispatch 2024-03-26T02:14:50.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.b", "id": [2, 1]}]: dispatch 2024-03-26T02:14:50.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.c", "id": [7, 1]}]: dispatch 2024-03-26T02:14:50.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.1b", "id": [7, 4]}]: dispatch 2024-03-26T02:14:50.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.1d", "id": [2, 0]}]: dispatch 2024-03-26T02:14:50.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:50 smithi072 ceph-mon[30177]: pgmap v911: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 299 B/s wr, 1 op/s 2024-03-26T02:14:50.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:14:50.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:14:50.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.2", "id": [2, 0]}]: dispatch 2024-03-26T02:14:50.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.7", "id": [3, 1]}]: dispatch 2024-03-26T02:14:50.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.b", "id": [2, 1]}]: dispatch 2024-03-26T02:14:50.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.c", "id": [7, 1]}]: dispatch 2024-03-26T02:14:50.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.1b", "id": [7, 4]}]: dispatch 2024-03-26T02:14:50.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.1d", "id": [2, 0]}]: dispatch 2024-03-26T02:14:50.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:50 smithi072 ceph-mon[41619]: pgmap v911: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 299 B/s wr, 1 op/s 2024-03-26T02:14:51.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:14:51 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:14:51.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:51 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.2", "id": [2, 0]}]': finished 2024-03-26T02:14:51.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:51 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.7", "id": [3, 1]}]': finished 2024-03-26T02:14:51.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:51 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.b", "id": [2, 1]}]': finished 2024-03-26T02:14:51.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:51 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.c", "id": [7, 1]}]': finished 2024-03-26T02:14:51.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:51 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.1b", "id": [7, 4]}]': finished 2024-03-26T02:14:51.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:51 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.1d", "id": [2, 0]}]': finished 2024-03-26T02:14:51.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:51 smithi173 ceph-mon[33839]: osdmap e590: 8 total, 8 up, 8 in 2024-03-26T02:14:51.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:51 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:14:51.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:51 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:14:51.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:51 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:14:51.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:51 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:14:51.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:51 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:14:51.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:51 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:14:51.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:51 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.2", "id": [2, 0]}]': finished 2024-03-26T02:14:51.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:51 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.7", "id": [3, 1]}]': finished 2024-03-26T02:14:51.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:51 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.b", "id": [2, 1]}]': finished 2024-03-26T02:14:51.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:51 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.c", "id": [7, 1]}]': finished 2024-03-26T02:14:51.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:51 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.1b", "id": [7, 4]}]': finished 2024-03-26T02:14:51.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:51 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.1d", "id": [2, 0]}]': finished 2024-03-26T02:14:51.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:51 smithi072 ceph-mon[30177]: osdmap e590: 8 total, 8 up, 8 in 2024-03-26T02:14:51.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:51 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:14:51.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:51 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:14:51.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:51 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:14:51.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:51 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:14:51.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:51 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:14:51.721 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:51 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:14:51.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:51 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.2", "id": [2, 0]}]': finished 2024-03-26T02:14:51.722 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:51 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.7", "id": [3, 1]}]': finished 2024-03-26T02:14:51.722 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:51 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.b", "id": [2, 1]}]': finished 2024-03-26T02:14:51.722 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:51 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.c", "id": [7, 1]}]': finished 2024-03-26T02:14:51.722 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:51 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.1b", "id": [7, 4]}]': finished 2024-03-26T02:14:51.722 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:51 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "315.1d", "id": [2, 0]}]': finished 2024-03-26T02:14:51.722 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:51 smithi072 ceph-mon[41619]: osdmap e590: 8 total, 8 up, 8 in 2024-03-26T02:14:51.722 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:51 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:14:51.722 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:51 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:14:51.722 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:51 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:14:51.722 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:51 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:14:51.723 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:51 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:14:51.723 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:51 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:14:52.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:52 smithi072 ceph-mon[30177]: osdmap e591: 8 total, 8 up, 8 in 2024-03-26T02:14:52.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:52 smithi072 ceph-mon[30177]: pgmap v914: 300 pgs: 6 peering, 294 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:14:52.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:14:52.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:14:52.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:52 smithi072 ceph-mon[41619]: osdmap e591: 8 total, 8 up, 8 in 2024-03-26T02:14:52.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:52 smithi072 ceph-mon[41619]: pgmap v914: 300 pgs: 6 peering, 294 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:14:52.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:14:52.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:14:52.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:52 smithi173 ceph-mon[33839]: osdmap e591: 8 total, 8 up, 8 in 2024-03-26T02:14:52.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:52 smithi173 ceph-mon[33839]: pgmap v914: 300 pgs: 6 peering, 294 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:14:52.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:14:52.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:14:55.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:54 smithi173 ceph-mon[33839]: pgmap v915: 300 pgs: 6 peering, 294 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:14:55.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:54 smithi072 ceph-mon[30177]: pgmap v915: 300 pgs: 6 peering, 294 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:14:55.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:54 smithi072 ceph-mon[41619]: pgmap v915: 300 pgs: 6 peering, 294 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:14:55.220 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:14:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:14:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:14:56.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:55 smithi173 ceph-mon[33839]: osdmap e592: 8 total, 8 up, 8 in 2024-03-26T02:14:56.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:55 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:14:56.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:55 smithi072 ceph-mon[30177]: osdmap e592: 8 total, 8 up, 8 in 2024-03-26T02:14:56.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:55 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:14:56.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:55 smithi072 ceph-mon[41619]: osdmap e592: 8 total, 8 up, 8 in 2024-03-26T02:14:56.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:55 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:14:57.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:56 smithi173 ceph-mon[33839]: pgmap v917: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T02:14:57.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:56 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:14:57.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:56 smithi173 ceph-mon[33839]: osdmap e593: 8 total, 8 up, 8 in 2024-03-26T02:14:57.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:56 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-116"}]: dispatch 2024-03-26T02:14:57.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:56 smithi072 ceph-mon[30177]: pgmap v917: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T02:14:57.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:56 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:14:57.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:56 smithi072 ceph-mon[30177]: osdmap e593: 8 total, 8 up, 8 in 2024-03-26T02:14:57.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:56 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-116"}]: dispatch 2024-03-26T02:14:57.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:56 smithi072 ceph-mon[41619]: pgmap v917: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T02:14:57.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:56 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:14:57.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:56 smithi072 ceph-mon[41619]: osdmap e593: 8 total, 8 up, 8 in 2024-03-26T02:14:57.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:56 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-116"}]: dispatch 2024-03-26T02:14:58.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:57 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:14:58.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:57 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-116"}]': finished 2024-03-26T02:14:58.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:57 smithi173 ceph-mon[33839]: osdmap e594: 8 total, 8 up, 8 in 2024-03-26T02:14:58.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:57 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:14:58.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:57 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-116"}]': finished 2024-03-26T02:14:58.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:57 smithi072 ceph-mon[30177]: osdmap e594: 8 total, 8 up, 8 in 2024-03-26T02:14:58.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:57 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:14:58.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:57 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-116"}]': finished 2024-03-26T02:14:58.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:57 smithi072 ceph-mon[41619]: osdmap e594: 8 total, 8 up, 8 in 2024-03-26T02:14:59.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:58 smithi173 ceph-mon[33839]: pgmap v920: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T02:14:59.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:58 smithi173 ceph-mon[33839]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:14:59.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:58 smithi173 ceph-mon[33839]: osdmap e595: 8 total, 8 up, 8 in 2024-03-26T02:14:59.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:58 smithi072 ceph-mon[41619]: pgmap v920: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T02:14:59.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:58 smithi072 ceph-mon[41619]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:14:59.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:58 smithi072 ceph-mon[41619]: osdmap e595: 8 total, 8 up, 8 in 2024-03-26T02:14:59.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:58 smithi072 ceph-mon[30177]: pgmap v920: 300 pgs: 300 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T02:14:59.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:58 smithi072 ceph-mon[30177]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:14:59.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:58 smithi072 ceph-mon[30177]: osdmap e595: 8 total, 8 up, 8 in 2024-03-26T02:15:00.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:59 smithi173 ceph-mon[33839]: osdmap e596: 8 total, 8 up, 8 in 2024-03-26T02:15:00.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:14:59 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:15:00.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:59 smithi072 ceph-mon[41619]: osdmap e596: 8 total, 8 up, 8 in 2024-03-26T02:15:00.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:14:59 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:15:00.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:59 smithi072 ceph-mon[30177]: osdmap e596: 8 total, 8 up, 8 in 2024-03-26T02:15:00.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:14:59 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:15:01.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:00 smithi173 ceph-mon[33839]: pgmap v923: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:15:01.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:15:01.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:00 smithi173 ceph-mon[33839]: osdmap e597: 8 total, 8 up, 8 in 2024-03-26T02:15:01.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:00 smithi173 ceph-mon[33839]: osdmap e598: 8 total, 8 up, 8 in 2024-03-26T02:15:01.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:15:01.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:00 smithi072 ceph-mon[41619]: pgmap v923: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:15:01.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:15:01.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:00 smithi072 ceph-mon[41619]: osdmap e597: 8 total, 8 up, 8 in 2024-03-26T02:15:01.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:00 smithi072 ceph-mon[41619]: osdmap e598: 8 total, 8 up, 8 in 2024-03-26T02:15:01.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:15:01.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:00 smithi072 ceph-mon[30177]: pgmap v923: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 605 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:15:01.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:15:01.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:00 smithi072 ceph-mon[30177]: osdmap e597: 8 total, 8 up, 8 in 2024-03-26T02:15:01.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:00 smithi072 ceph-mon[30177]: osdmap e598: 8 total, 8 up, 8 in 2024-03-26T02:15:01.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:15:01.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:15:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:15:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:01 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:15:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:15:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:01 smithi072 ceph-mon[30177]: osdmap e599: 8 total, 8 up, 8 in 2024-03-26T02:15:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-118"}]: dispatch 2024-03-26T02:15:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:01 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:15:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:15:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:01 smithi072 ceph-mon[41619]: osdmap e599: 8 total, 8 up, 8 in 2024-03-26T02:15:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-118"}]: dispatch 2024-03-26T02:15:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:01 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:15:02.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:15:02.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:01 smithi173 ceph-mon[33839]: osdmap e599: 8 total, 8 up, 8 in 2024-03-26T02:15:02.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-118"}]: dispatch 2024-03-26T02:15:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:02 smithi072 ceph-mon[30177]: pgmap v926: 300 pgs: 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:15:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-118"}]': finished 2024-03-26T02:15:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:02 smithi072 ceph-mon[30177]: osdmap e600: 8 total, 8 up, 8 in 2024-03-26T02:15:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-118", "mode": "writeback"}]: dispatch 2024-03-26T02:15:03.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:02 smithi072 ceph-mon[41619]: pgmap v926: 300 pgs: 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:15:03.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-118"}]': finished 2024-03-26T02:15:03.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:02 smithi072 ceph-mon[41619]: osdmap e600: 8 total, 8 up, 8 in 2024-03-26T02:15:03.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-118", "mode": "writeback"}]: dispatch 2024-03-26T02:15:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:02 smithi173 ceph-mon[33839]: pgmap v926: 300 pgs: 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:15:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-118"}]': finished 2024-03-26T02:15:03.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:02 smithi173 ceph-mon[33839]: osdmap e600: 8 total, 8 up, 8 in 2024-03-26T02:15:03.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-118", "mode": "writeback"}]: dispatch 2024-03-26T02:15:04.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:03 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:15:04.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-118", "mode": "writeback"}]': finished 2024-03-26T02:15:04.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:03 smithi072 ceph-mon[30177]: osdmap e601: 8 total, 8 up, 8 in 2024-03-26T02:15:04.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:03 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:15:04.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-118", "mode": "writeback"}]': finished 2024-03-26T02:15:04.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:03 smithi072 ceph-mon[41619]: osdmap e601: 8 total, 8 up, 8 in 2024-03-26T02:15:04.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:03 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:15:04.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-118", "mode": "writeback"}]': finished 2024-03-26T02:15:04.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:03 smithi173 ceph-mon[33839]: osdmap e601: 8 total, 8 up, 8 in 2024-03-26T02:15:05.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:15:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:15:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:15:05.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:04 smithi072 ceph-mon[41619]: pgmap v929: 300 pgs: 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:05.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:04 smithi072 ceph-mon[41619]: osdmap e602: 8 total, 8 up, 8 in 2024-03-26T02:15:05.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:04 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:15:05.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:04 smithi072 ceph-mon[30177]: pgmap v929: 300 pgs: 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:05.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:04 smithi072 ceph-mon[30177]: osdmap e602: 8 total, 8 up, 8 in 2024-03-26T02:15:05.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:04 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:15:05.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:04 smithi173 ceph-mon[33839]: pgmap v929: 300 pgs: 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:05.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:04 smithi173 ceph-mon[33839]: osdmap e602: 8 total, 8 up, 8 in 2024-03-26T02:15:05.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:04 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:15:06.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:05 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:06.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:15:06.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:05 smithi072 ceph-mon[30177]: osdmap e603: 8 total, 8 up, 8 in 2024-03-26T02:15:06.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-118"}]: dispatch 2024-03-26T02:15:06.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:05 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:06.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:15:06.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:05 smithi072 ceph-mon[41619]: osdmap e603: 8 total, 8 up, 8 in 2024-03-26T02:15:06.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-118"}]: dispatch 2024-03-26T02:15:06.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:05 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:06.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:15:06.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:05 smithi173 ceph-mon[33839]: osdmap e603: 8 total, 8 up, 8 in 2024-03-26T02:15:06.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-118"}]: dispatch 2024-03-26T02:15:07.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:06 smithi072 ceph-mon[30177]: pgmap v932: 300 pgs: 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:07.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:06 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:15:07.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-118"}]': finished 2024-03-26T02:15:07.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:06 smithi072 ceph-mon[30177]: osdmap e604: 8 total, 8 up, 8 in 2024-03-26T02:15:07.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:06 smithi072 ceph-mon[41619]: pgmap v932: 300 pgs: 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:07.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:06 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:15:07.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-118"}]': finished 2024-03-26T02:15:07.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:06 smithi072 ceph-mon[41619]: osdmap e604: 8 total, 8 up, 8 in 2024-03-26T02:15:07.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:06 smithi173 ceph-mon[33839]: pgmap v932: 300 pgs: 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:07.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:06 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:15:07.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-118"}]': finished 2024-03-26T02:15:07.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:06 smithi173 ceph-mon[33839]: osdmap e604: 8 total, 8 up, 8 in 2024-03-26T02:15:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:15:08.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:15:08.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:08 smithi173 ceph-mon[33839]: osdmap e605: 8 total, 8 up, 8 in 2024-03-26T02:15:08.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:15:08.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:15:08.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:08 smithi072 ceph-mon[30177]: osdmap e605: 8 total, 8 up, 8 in 2024-03-26T02:15:08.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:15:08.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:15:08.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:08 smithi072 ceph-mon[41619]: osdmap e605: 8 total, 8 up, 8 in 2024-03-26T02:15:09.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:09 smithi173 ceph-mon[33839]: pgmap v935: 300 pgs: 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:09.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:09 smithi173 ceph-mon[33839]: osdmap e606: 8 total, 8 up, 8 in 2024-03-26T02:15:09.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:15:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:09 smithi072 ceph-mon[30177]: pgmap v935: 300 pgs: 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:09 smithi072 ceph-mon[30177]: osdmap e606: 8 total, 8 up, 8 in 2024-03-26T02:15:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:15:09.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:09 smithi072 ceph-mon[41619]: pgmap v935: 300 pgs: 300 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:09.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:09 smithi072 ceph-mon[41619]: osdmap e606: 8 total, 8 up, 8 in 2024-03-26T02:15:09.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:15:11.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:10 smithi173 ceph-mon[33839]: pgmap v938: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:15:11.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:15:11.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:10 smithi173 ceph-mon[33839]: osdmap e607: 8 total, 8 up, 8 in 2024-03-26T02:15:11.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:15:11.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:10 smithi173 ceph-mon[33839]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:11.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:15:11.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:10 smithi173 ceph-mon[33839]: osdmap e608: 8 total, 8 up, 8 in 2024-03-26T02:15:11.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-120"}]: dispatch 2024-03-26T02:15:11.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:10 smithi072 ceph-mon[30177]: pgmap v938: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:15:11.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:15:11.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:10 smithi072 ceph-mon[30177]: osdmap e607: 8 total, 8 up, 8 in 2024-03-26T02:15:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:15:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:10 smithi072 ceph-mon[30177]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:15:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:10 smithi072 ceph-mon[30177]: osdmap e608: 8 total, 8 up, 8 in 2024-03-26T02:15:11.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-120"}]: dispatch 2024-03-26T02:15:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:10 smithi072 ceph-mon[41619]: pgmap v938: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 624 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:15:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:15:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:10 smithi072 ceph-mon[41619]: osdmap e607: 8 total, 8 up, 8 in 2024-03-26T02:15:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:15:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:10 smithi072 ceph-mon[41619]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:15:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:10 smithi072 ceph-mon[41619]: osdmap e608: 8 total, 8 up, 8 in 2024-03-26T02:15:11.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-120"}]: dispatch 2024-03-26T02:15:11.335 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:15:11 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:15:12.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:11 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:15:12.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:11 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-120"}]': finished 2024-03-26T02:15:12.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:11 smithi173 ceph-mon[33839]: osdmap e609: 8 total, 8 up, 8 in 2024-03-26T02:15:12.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:11 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-120", "mode": "writeback"}]: dispatch 2024-03-26T02:15:12.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:11 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:15:12.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:11 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-120"}]': finished 2024-03-26T02:15:12.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:11 smithi072 ceph-mon[30177]: osdmap e609: 8 total, 8 up, 8 in 2024-03-26T02:15:12.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:11 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-120", "mode": "writeback"}]: dispatch 2024-03-26T02:15:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:11 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:15:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:11 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-120"}]': finished 2024-03-26T02:15:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:11 smithi072 ceph-mon[41619]: osdmap e609: 8 total, 8 up, 8 in 2024-03-26T02:15:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:11 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-120", "mode": "writeback"}]: dispatch 2024-03-26T02:15:13.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:12 smithi173 ceph-mon[33839]: pgmap v941: 300 pgs: 300 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:15:13.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:12 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:15:13.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-120", "mode": "writeback"}]': finished 2024-03-26T02:15:13.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:12 smithi173 ceph-mon[33839]: osdmap e610: 8 total, 8 up, 8 in 2024-03-26T02:15:13.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:12 smithi072 ceph-mon[30177]: pgmap v941: 300 pgs: 300 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:15:13.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:12 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:15:13.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-120", "mode": "writeback"}]': finished 2024-03-26T02:15:13.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:12 smithi072 ceph-mon[30177]: osdmap e610: 8 total, 8 up, 8 in 2024-03-26T02:15:13.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:12 smithi072 ceph-mon[41619]: pgmap v941: 300 pgs: 300 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:15:13.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:12 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:15:13.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-120", "mode": "writeback"}]': finished 2024-03-26T02:15:13.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:12 smithi072 ceph-mon[41619]: osdmap e610: 8 total, 8 up, 8 in 2024-03-26T02:15:14.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:13 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:15:14.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:13 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:15:14.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:13 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:15:15.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:14 smithi173 ceph-mon[33839]: pgmap v944: 300 pgs: 300 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:15.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:14 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:15:15.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:14 smithi173 ceph-mon[33839]: osdmap e611: 8 total, 8 up, 8 in 2024-03-26T02:15:15.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:14 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-120"}]: dispatch 2024-03-26T02:15:15.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:15:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:15:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:15:15.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:14 smithi072 ceph-mon[30177]: pgmap v944: 300 pgs: 300 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:15.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:14 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:15:15.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:14 smithi072 ceph-mon[30177]: osdmap e611: 8 total, 8 up, 8 in 2024-03-26T02:15:15.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:14 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-120"}]: dispatch 2024-03-26T02:15:15.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:14 smithi072 ceph-mon[41619]: pgmap v944: 300 pgs: 300 active+clean; 582 KiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:15.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:14 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:15:15.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:14 smithi072 ceph-mon[41619]: osdmap e611: 8 total, 8 up, 8 in 2024-03-26T02:15:15.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:14 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-120"}]: dispatch 2024-03-26T02:15:16.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:15 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:15:16.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-120"}]': finished 2024-03-26T02:15:16.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:15 smithi173 ceph-mon[33839]: osdmap e612: 8 total, 8 up, 8 in 2024-03-26T02:15:16.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:15 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:15 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:15:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-120"}]': finished 2024-03-26T02:15:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:15 smithi072 ceph-mon[30177]: osdmap e612: 8 total, 8 up, 8 in 2024-03-26T02:15:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:15 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:15 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:15:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-120"}]': finished 2024-03-26T02:15:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:15 smithi072 ceph-mon[41619]: osdmap e612: 8 total, 8 up, 8 in 2024-03-26T02:15:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:15 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:17.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:16 smithi173 ceph-mon[33839]: pgmap v947: 300 pgs: 300 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T02:15:17.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:16 smithi173 ceph-mon[33839]: osdmap e613: 8 total, 8 up, 8 in 2024-03-26T02:15:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:16 smithi072 ceph-mon[30177]: pgmap v947: 300 pgs: 300 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T02:15:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:16 smithi072 ceph-mon[30177]: osdmap e613: 8 total, 8 up, 8 in 2024-03-26T02:15:17.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:16 smithi072 ceph-mon[41619]: pgmap v947: 300 pgs: 300 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T02:15:17.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:16 smithi072 ceph-mon[41619]: osdmap e613: 8 total, 8 up, 8 in 2024-03-26T02:15:18.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:17 smithi072 ceph-mon[30177]: osdmap e614: 8 total, 8 up, 8 in 2024-03-26T02:15:18.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:17 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:15:18.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:17 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:15:18.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:17 smithi072 ceph-mon[30177]: osdmap e615: 8 total, 8 up, 8 in 2024-03-26T02:15:18.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:17 smithi072 ceph-mon[41619]: osdmap e614: 8 total, 8 up, 8 in 2024-03-26T02:15:18.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:17 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:15:18.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:17 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:15:18.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:17 smithi072 ceph-mon[41619]: osdmap e615: 8 total, 8 up, 8 in 2024-03-26T02:15:18.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:17 smithi173 ceph-mon[33839]: osdmap e614: 8 total, 8 up, 8 in 2024-03-26T02:15:18.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:17 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:15:18.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:17 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:15:18.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:17 smithi173 ceph-mon[33839]: osdmap e615: 8 total, 8 up, 8 in 2024-03-26T02:15:19.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:18 smithi072 ceph-mon[30177]: pgmap v950: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:19.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:15:19.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:15:19.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:18 smithi072 ceph-mon[30177]: osdmap e616: 8 total, 8 up, 8 in 2024-03-26T02:15:19.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-122"}]: dispatch 2024-03-26T02:15:19.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:18 smithi072 ceph-mon[41619]: pgmap v950: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:19.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:15:19.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:15:19.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:18 smithi072 ceph-mon[41619]: osdmap e616: 8 total, 8 up, 8 in 2024-03-26T02:15:19.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-122"}]: dispatch 2024-03-26T02:15:19.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:18 smithi173 ceph-mon[33839]: pgmap v950: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:19.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:15:19.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:15:19.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:18 smithi173 ceph-mon[33839]: osdmap e616: 8 total, 8 up, 8 in 2024-03-26T02:15:19.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-122"}]: dispatch 2024-03-26T02:15:21.184 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:20 smithi173 ceph-mon[33839]: pgmap v953: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:15:21.184 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-122"}]': finished 2024-03-26T02:15:21.184 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:20 smithi173 ceph-mon[33839]: osdmap e617: 8 total, 8 up, 8 in 2024-03-26T02:15:21.185 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-122", "mode": "writeback"}]: dispatch 2024-03-26T02:15:21.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:20 smithi072 ceph-mon[30177]: pgmap v953: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:15:21.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-122"}]': finished 2024-03-26T02:15:21.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:20 smithi072 ceph-mon[30177]: osdmap e617: 8 total, 8 up, 8 in 2024-03-26T02:15:21.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-122", "mode": "writeback"}]: dispatch 2024-03-26T02:15:21.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:20 smithi072 ceph-mon[41619]: pgmap v953: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:15:21.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-122"}]': finished 2024-03-26T02:15:21.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:20 smithi072 ceph-mon[41619]: osdmap e617: 8 total, 8 up, 8 in 2024-03-26T02:15:21.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-122", "mode": "writeback"}]: dispatch 2024-03-26T02:15:21.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:15:21 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:15:22.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:21 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:15:22.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-122", "mode": "writeback"}]': finished 2024-03-26T02:15:22.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:21 smithi173 ceph-mon[33839]: osdmap e618: 8 total, 8 up, 8 in 2024-03-26T02:15:22.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:15:22.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:21 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:15:22.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:21 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:15:22.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-122", "mode": "writeback"}]': finished 2024-03-26T02:15:22.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:21 smithi072 ceph-mon[30177]: osdmap e618: 8 total, 8 up, 8 in 2024-03-26T02:15:22.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:15:22.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:21 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:15:22.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:21 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:15:22.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-122", "mode": "writeback"}]': finished 2024-03-26T02:15:22.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:21 smithi072 ceph-mon[41619]: osdmap e618: 8 total, 8 up, 8 in 2024-03-26T02:15:22.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:15:22.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:21 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:15:23.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:22 smithi072 ceph-mon[30177]: pgmap v956: 300 pgs: 300 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:23.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:22 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:15:23.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:22 smithi072 ceph-mon[30177]: osdmap e619: 8 total, 8 up, 8 in 2024-03-26T02:15:23.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:22 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-122"}]: dispatch 2024-03-26T02:15:23.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:22 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:15:23.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:22 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:15:23.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:22 smithi072 ceph-mon[41619]: pgmap v956: 300 pgs: 300 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:23.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:22 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:15:23.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:22 smithi072 ceph-mon[41619]: osdmap e619: 8 total, 8 up, 8 in 2024-03-26T02:15:23.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:22 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-122"}]: dispatch 2024-03-26T02:15:23.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:22 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:15:23.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:22 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:15:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:22 smithi173 ceph-mon[33839]: pgmap v956: 300 pgs: 300 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:22 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:15:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:22 smithi173 ceph-mon[33839]: osdmap e619: 8 total, 8 up, 8 in 2024-03-26T02:15:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:22 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-122"}]: dispatch 2024-03-26T02:15:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:22 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:15:23.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:22 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:15:24.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:23 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:15:24.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:23 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-122"}]': finished 2024-03-26T02:15:24.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:23 smithi072 ceph-mon[41619]: osdmap e620: 8 total, 8 up, 8 in 2024-03-26T02:15:24.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:23 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:15:24.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:23 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-122"}]': finished 2024-03-26T02:15:24.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:23 smithi072 ceph-mon[30177]: osdmap e620: 8 total, 8 up, 8 in 2024-03-26T02:15:24.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:23 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:15:24.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:23 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-122"}]': finished 2024-03-26T02:15:24.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:23 smithi173 ceph-mon[33839]: osdmap e620: 8 total, 8 up, 8 in 2024-03-26T02:15:25.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:15:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:15:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:15:25.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:24 smithi072 ceph-mon[41619]: pgmap v959: 300 pgs: 300 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:25.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:24 smithi072 ceph-mon[41619]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:25.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:24 smithi072 ceph-mon[41619]: osdmap e621: 8 total, 8 up, 8 in 2024-03-26T02:15:25.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:24 smithi072 ceph-mon[30177]: pgmap v959: 300 pgs: 300 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:25.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:24 smithi072 ceph-mon[30177]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:25.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:24 smithi072 ceph-mon[30177]: osdmap e621: 8 total, 8 up, 8 in 2024-03-26T02:15:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:24 smithi173 ceph-mon[33839]: pgmap v959: 300 pgs: 300 active+clean; 582 KiB data, 630 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:24 smithi173 ceph-mon[33839]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:24 smithi173 ceph-mon[33839]: osdmap e621: 8 total, 8 up, 8 in 2024-03-26T02:15:26.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:25 smithi072 ceph-mon[30177]: osdmap e622: 8 total, 8 up, 8 in 2024-03-26T02:15:26.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:15:26.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:25 smithi072 ceph-mon[41619]: osdmap e622: 8 total, 8 up, 8 in 2024-03-26T02:15:26.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:15:26.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:25 smithi173 ceph-mon[33839]: osdmap e622: 8 total, 8 up, 8 in 2024-03-26T02:15:26.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:15:27.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:26 smithi072 ceph-mon[30177]: pgmap v962: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:27.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:26 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:15:27.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:26 smithi072 ceph-mon[30177]: osdmap e623: 8 total, 8 up, 8 in 2024-03-26T02:15:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:26 smithi072 ceph-mon[41619]: pgmap v962: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:26 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:15:27.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:26 smithi072 ceph-mon[41619]: osdmap e623: 8 total, 8 up, 8 in 2024-03-26T02:15:27.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:26 smithi173 ceph-mon[33839]: pgmap v962: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:27.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:26 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:15:27.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:26 smithi173 ceph-mon[33839]: osdmap e623: 8 total, 8 up, 8 in 2024-03-26T02:15:28.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:27 smithi072 ceph-mon[30177]: osdmap e624: 8 total, 8 up, 8 in 2024-03-26T02:15:28.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:27 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:15:28.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:27 smithi072 ceph-mon[41619]: osdmap e624: 8 total, 8 up, 8 in 2024-03-26T02:15:28.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:27 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:15:28.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:27 smithi173 ceph-mon[33839]: osdmap e624: 8 total, 8 up, 8 in 2024-03-26T02:15:28.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:27 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:15:29.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:28 smithi072 ceph-mon[30177]: pgmap v965: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:29.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:28 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:15:29.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:28 smithi072 ceph-mon[30177]: osdmap e625: 8 total, 8 up, 8 in 2024-03-26T02:15:29.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:28 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-124"}]: dispatch 2024-03-26T02:15:29.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:28 smithi072 ceph-mon[41619]: pgmap v965: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:29.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:28 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:15:29.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:28 smithi072 ceph-mon[41619]: osdmap e625: 8 total, 8 up, 8 in 2024-03-26T02:15:29.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:28 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-124"}]: dispatch 2024-03-26T02:15:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:28 smithi173 ceph-mon[33839]: pgmap v965: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:28 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:15:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:28 smithi173 ceph-mon[33839]: osdmap e625: 8 total, 8 up, 8 in 2024-03-26T02:15:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:28 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-124"}]: dispatch 2024-03-26T02:15:30.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:29 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-124"}]': finished 2024-03-26T02:15:30.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:29 smithi072 ceph-mon[30177]: osdmap e626: 8 total, 8 up, 8 in 2024-03-26T02:15:30.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:29 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-124", "mode": "writeback"}]: dispatch 2024-03-26T02:15:30.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:29 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:15:30.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:29 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-124", "mode": "writeback"}]': finished 2024-03-26T02:15:30.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:29 smithi072 ceph-mon[30177]: osdmap e627: 8 total, 8 up, 8 in 2024-03-26T02:15:30.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:29 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-124"}]': finished 2024-03-26T02:15:30.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:29 smithi072 ceph-mon[41619]: osdmap e626: 8 total, 8 up, 8 in 2024-03-26T02:15:30.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:29 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-124", "mode": "writeback"}]: dispatch 2024-03-26T02:15:30.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:29 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:15:30.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:29 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-124", "mode": "writeback"}]': finished 2024-03-26T02:15:30.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:29 smithi072 ceph-mon[41619]: osdmap e627: 8 total, 8 up, 8 in 2024-03-26T02:15:30.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:29 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-124"}]': finished 2024-03-26T02:15:30.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:29 smithi173 ceph-mon[33839]: osdmap e626: 8 total, 8 up, 8 in 2024-03-26T02:15:30.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:29 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-124", "mode": "writeback"}]: dispatch 2024-03-26T02:15:30.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:29 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:15:30.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:29 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-124", "mode": "writeback"}]': finished 2024-03-26T02:15:30.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:29 smithi173 ceph-mon[33839]: osdmap e627: 8 total, 8 up, 8 in 2024-03-26T02:15:31.193 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:30 smithi173 ceph-mon[33839]: pgmap v968: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:15:31.193 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:30 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:31.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:30 smithi072 ceph-mon[30177]: pgmap v968: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:15:31.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:30 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:31.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:30 smithi072 ceph-mon[41619]: pgmap v968: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 631 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:15:31.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:30 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:31.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:15:31 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:15:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:31 smithi072 ceph-mon[30177]: osdmap e628: 8 total, 8 up, 8 in 2024-03-26T02:15:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:31 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:15:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:31 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:15:32.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:31 smithi072 ceph-mon[41619]: osdmap e628: 8 total, 8 up, 8 in 2024-03-26T02:15:32.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:31 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:15:32.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:31 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:15:32.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:31 smithi173 ceph-mon[33839]: osdmap e628: 8 total, 8 up, 8 in 2024-03-26T02:15:32.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:31 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:15:32.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:31 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:15:33.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:32 smithi072 ceph-mon[30177]: pgmap v971: 300 pgs: 300 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-26T02:15:33.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:15:33.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:32 smithi072 ceph-mon[30177]: osdmap e629: 8 total, 8 up, 8 in 2024-03-26T02:15:33.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-124"}]: dispatch 2024-03-26T02:15:33.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:32 smithi072 ceph-mon[41619]: pgmap v971: 300 pgs: 300 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-26T02:15:33.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:15:33.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:32 smithi072 ceph-mon[41619]: osdmap e629: 8 total, 8 up, 8 in 2024-03-26T02:15:33.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-124"}]: dispatch 2024-03-26T02:15:33.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:32 smithi173 ceph-mon[33839]: pgmap v971: 300 pgs: 300 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-26T02:15:33.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:15:33.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:32 smithi173 ceph-mon[33839]: osdmap e629: 8 total, 8 up, 8 in 2024-03-26T02:15:33.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-124"}]: dispatch 2024-03-26T02:15:34.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:33 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:15:34.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:33 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-124"}]': finished 2024-03-26T02:15:34.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:33 smithi072 ceph-mon[30177]: osdmap e630: 8 total, 8 up, 8 in 2024-03-26T02:15:34.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:33 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:15:34.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:33 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-124"}]': finished 2024-03-26T02:15:34.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:33 smithi072 ceph-mon[41619]: osdmap e630: 8 total, 8 up, 8 in 2024-03-26T02:15:34.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:33 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:15:34.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:33 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-124"}]': finished 2024-03-26T02:15:34.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:33 smithi173 ceph-mon[33839]: osdmap e630: 8 total, 8 up, 8 in 2024-03-26T02:15:35.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:34 smithi072 ceph-mon[30177]: pgmap v974: 300 pgs: 300 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-26T02:15:35.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:34 smithi072 ceph-mon[30177]: osdmap e631: 8 total, 8 up, 8 in 2024-03-26T02:15:35.220 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:15:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:15:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:15:35.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:34 smithi072 ceph-mon[41619]: pgmap v974: 300 pgs: 300 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-26T02:15:35.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:34 smithi072 ceph-mon[41619]: osdmap e631: 8 total, 8 up, 8 in 2024-03-26T02:15:35.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:34 smithi173 ceph-mon[33839]: pgmap v974: 300 pgs: 300 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-26T02:15:35.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:34 smithi173 ceph-mon[33839]: osdmap e631: 8 total, 8 up, 8 in 2024-03-26T02:15:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:35 smithi072 ceph-mon[30177]: osdmap e632: 8 total, 8 up, 8 in 2024-03-26T02:15:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:35 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:15:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:35 smithi072 ceph-mon[30177]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:36.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:35 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:15:36.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:35 smithi072 ceph-mon[30177]: osdmap e633: 8 total, 8 up, 8 in 2024-03-26T02:15:36.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:35 smithi072 ceph-mon[41619]: osdmap e632: 8 total, 8 up, 8 in 2024-03-26T02:15:36.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:35 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:15:36.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:35 smithi072 ceph-mon[41619]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:36.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:35 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:15:36.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:35 smithi072 ceph-mon[41619]: osdmap e633: 8 total, 8 up, 8 in 2024-03-26T02:15:36.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:35 smithi173 ceph-mon[33839]: osdmap e632: 8 total, 8 up, 8 in 2024-03-26T02:15:36.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:35 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:15:36.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:35 smithi173 ceph-mon[33839]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:36.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:35 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:15:36.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:35 smithi173 ceph-mon[33839]: osdmap e633: 8 total, 8 up, 8 in 2024-03-26T02:15:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:36 smithi072 ceph-mon[30177]: pgmap v977: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:15:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:36 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:15:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:36 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:15:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:36 smithi072 ceph-mon[30177]: osdmap e634: 8 total, 8 up, 8 in 2024-03-26T02:15:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:36 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-126"}]: dispatch 2024-03-26T02:15:37.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:36 smithi072 ceph-mon[41619]: pgmap v977: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:15:37.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:36 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:15:37.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:36 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:15:37.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:36 smithi072 ceph-mon[41619]: osdmap e634: 8 total, 8 up, 8 in 2024-03-26T02:15:37.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:36 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-126"}]: dispatch 2024-03-26T02:15:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:36 smithi173 ceph-mon[33839]: pgmap v977: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:15:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:36 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:15:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:36 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:15:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:36 smithi173 ceph-mon[33839]: osdmap e634: 8 total, 8 up, 8 in 2024-03-26T02:15:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:36 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-126"}]: dispatch 2024-03-26T02:15:38.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:38 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:15:38.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:38 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:15:38.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:38 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-126"}]': finished 2024-03-26T02:15:38.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:38 smithi173 ceph-mon[33839]: osdmap e635: 8 total, 8 up, 8 in 2024-03-26T02:15:38.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:38 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-126", "mode": "writeback"}]: dispatch 2024-03-26T02:15:38.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:38 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:15:38.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:38 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:15:38.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:38 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-126"}]': finished 2024-03-26T02:15:38.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:38 smithi072 ceph-mon[30177]: osdmap e635: 8 total, 8 up, 8 in 2024-03-26T02:15:38.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:38 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-126", "mode": "writeback"}]: dispatch 2024-03-26T02:15:38.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:38 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:15:38.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:38 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:15:38.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:38 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-126"}]': finished 2024-03-26T02:15:38.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:38 smithi072 ceph-mon[41619]: osdmap e635: 8 total, 8 up, 8 in 2024-03-26T02:15:38.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:38 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-126", "mode": "writeback"}]: dispatch 2024-03-26T02:15:39.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:39 smithi173 ceph-mon[33839]: pgmap v980: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:15:39.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:39 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:15:39.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-126", "mode": "writeback"}]': finished 2024-03-26T02:15:39.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:39 smithi173 ceph-mon[33839]: osdmap e636: 8 total, 8 up, 8 in 2024-03-26T02:15:39.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:15:39.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:39 smithi072 ceph-mon[30177]: pgmap v980: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:15:39.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:39 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:15:39.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-126", "mode": "writeback"}]': finished 2024-03-26T02:15:39.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:39 smithi072 ceph-mon[30177]: osdmap e636: 8 total, 8 up, 8 in 2024-03-26T02:15:39.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:15:39.723 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:39 smithi072 ceph-mon[41619]: pgmap v980: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:15:39.723 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:39 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:15:39.723 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-126", "mode": "writeback"}]': finished 2024-03-26T02:15:39.723 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:39 smithi072 ceph-mon[41619]: osdmap e636: 8 total, 8 up, 8 in 2024-03-26T02:15:39.723 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:15:41.196 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:40 smithi173 ceph-mon[33839]: pgmap v983: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:15:41.196 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:15:41.196 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:40 smithi173 ceph-mon[33839]: osdmap e637: 8 total, 8 up, 8 in 2024-03-26T02:15:41.196 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-126"}]: dispatch 2024-03-26T02:15:41.196 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:40 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:41.196 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:40 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:15:41.196 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-126"}]': finished 2024-03-26T02:15:41.196 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:40 smithi173 ceph-mon[33839]: osdmap e638: 8 total, 8 up, 8 in 2024-03-26T02:15:41.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:40 smithi072 ceph-mon[30177]: pgmap v983: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:15:41.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:15:41.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:40 smithi072 ceph-mon[30177]: osdmap e637: 8 total, 8 up, 8 in 2024-03-26T02:15:41.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-126"}]: dispatch 2024-03-26T02:15:41.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:40 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:41.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:40 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:15:41.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-126"}]': finished 2024-03-26T02:15:41.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:40 smithi072 ceph-mon[30177]: osdmap e638: 8 total, 8 up, 8 in 2024-03-26T02:15:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:40 smithi072 ceph-mon[41619]: pgmap v983: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:15:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:15:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:40 smithi072 ceph-mon[41619]: osdmap e637: 8 total, 8 up, 8 in 2024-03-26T02:15:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-126"}]: dispatch 2024-03-26T02:15:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:40 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:40 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:15:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-126"}]': finished 2024-03-26T02:15:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:40 smithi072 ceph-mon[41619]: osdmap e638: 8 total, 8 up, 8 in 2024-03-26T02:15:41.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:15:41 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:15:41.752 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T02:15:41.752 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (9064 ms) 2024-03-26T02:15:41.752 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-03-26T02:15:41.752 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T02:15:41.752 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6047 ms) 2024-03-26T02:15:41.752 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-03-26T02:15:41.752 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T02:15:41.753 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5037 ms) 2024-03-26T02:15:41.753 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-03-26T02:15:41.753 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T02:15:41.753 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (25466 ms) 2024-03-26T02:15:41.753 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-03-26T02:15:41.753 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T02:15:41.753 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (14066 ms) 2024-03-26T02:15:41.753 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-03-26T02:15:41.753 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12087 ms) 2024-03-26T02:15:41.753 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-03-26T02:15:41.753 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-03-26T02:15:41.753 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: 00000010 69 73 |is| 2024-03-26T02:15:41.753 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: 00000012 2024-03-26T02:15:41.753 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12086 ms) 2024-03-26T02:15:41.753 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-03-26T02:15:41.754 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T02:15:41.754 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6045 ms) 2024-03-26T02:15:41.754 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (557953 ms total) 2024-03-26T02:15:41.754 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: 2024-03-26T02:15:41.754 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-03-26T02:15:41.754 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-03-26T02:15:41.754 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1012 ms) 2024-03-26T02:15:41.754 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-03-26T02:15:41.754 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (11066 ms) 2024-03-26T02:15:41.754 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-03-26T02:15:41.754 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18073 ms) 2024-03-26T02:15:41.754 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-03-26T02:15:41.754 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (2 ms) 2024-03-26T02:15:41.754 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30153 ms total) 2024-03-26T02:15:41.755 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: 2024-03-26T02:15:41.755 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-03-26T02:15:41.755 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-03-26T02:15:41.755 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7057 ms) 2024-03-26T02:15:41.755 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-03-26T02:15:41.755 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8049 ms) 2024-03-26T02:15:41.755 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-03-26T02:15:41.755 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: waiting for scrub... 2024-03-26T02:15:41.755 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: done waiting 2024-03-26T02:15:41.755 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (24346 ms) 2024-03-26T02:15:41.755 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-03-26T02:15:41.755 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (9965 ms) 2024-03-26T02:15:41.755 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-03-26T02:15:41.755 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8033 ms) 2024-03-26T02:15:41.755 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-03-26T02:15:41.756 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8061 ms) 2024-03-26T02:15:41.756 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-03-26T02:15:41.756 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10074 ms) 2024-03-26T02:15:41.756 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-03-26T02:15:42.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:42 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:15:42.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:42 smithi173 ceph-mon[33839]: osdmap e639: 8 total, 8 up, 8 in 2024-03-26T02:15:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:42 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:15:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:42 smithi072 ceph-mon[30177]: osdmap e639: 8 total, 8 up, 8 in 2024-03-26T02:15:42.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:42 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:15:42.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:42 smithi072 ceph-mon[41619]: osdmap e639: 8 total, 8 up, 8 in 2024-03-26T02:15:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:43 smithi173 ceph-mon[33839]: pgmap v986: 300 pgs: 300 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T02:15:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:43 smithi173 ceph-mon[33839]: osdmap e640: 8 total, 8 up, 8 in 2024-03-26T02:15:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:43 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:15:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:43 smithi072 ceph-mon[30177]: pgmap v986: 300 pgs: 300 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T02:15:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:43 smithi072 ceph-mon[30177]: osdmap e640: 8 total, 8 up, 8 in 2024-03-26T02:15:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:43 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:15:43.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:43 smithi072 ceph-mon[41619]: pgmap v986: 300 pgs: 300 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T02:15:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:43 smithi072 ceph-mon[41619]: osdmap e640: 8 total, 8 up, 8 in 2024-03-26T02:15:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:43 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:15:45.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:44 smithi173 ceph-mon[33839]: pgmap v989: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:45.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:44 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:15:45.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:44 smithi173 ceph-mon[33839]: osdmap e641: 8 total, 8 up, 8 in 2024-03-26T02:15:45.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:44 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:15:45.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:44 smithi072 ceph-mon[30177]: pgmap v989: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:45.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:44 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:15:45.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:44 smithi072 ceph-mon[30177]: osdmap e641: 8 total, 8 up, 8 in 2024-03-26T02:15:45.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:44 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:15:45.220 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:15:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:15:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:15:45.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:44 smithi072 ceph-mon[41619]: pgmap v989: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:15:45.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:44 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:15:45.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:44 smithi072 ceph-mon[41619]: osdmap e641: 8 total, 8 up, 8 in 2024-03-26T02:15:45.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:44 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:15:46.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:45 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:15:46.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:45 smithi173 ceph-mon[33839]: osdmap e642: 8 total, 8 up, 8 in 2024-03-26T02:15:46.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:45 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-128"}]: dispatch 2024-03-26T02:15:46.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:45 smithi173 ceph-mon[33839]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:46.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:45 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-128"}]': finished 2024-03-26T02:15:46.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:45 smithi173 ceph-mon[33839]: osdmap e643: 8 total, 8 up, 8 in 2024-03-26T02:15:46.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:45 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-128", "mode": "writeback"}]: dispatch 2024-03-26T02:15:46.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:45 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:15:46.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:45 smithi072 ceph-mon[30177]: osdmap e642: 8 total, 8 up, 8 in 2024-03-26T02:15:46.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:45 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-128"}]: dispatch 2024-03-26T02:15:46.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:45 smithi072 ceph-mon[30177]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:46.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:45 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-128"}]': finished 2024-03-26T02:15:46.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:45 smithi072 ceph-mon[30177]: osdmap e643: 8 total, 8 up, 8 in 2024-03-26T02:15:46.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:45 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-128", "mode": "writeback"}]: dispatch 2024-03-26T02:15:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:45 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:15:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:45 smithi072 ceph-mon[41619]: osdmap e642: 8 total, 8 up, 8 in 2024-03-26T02:15:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:45 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-128"}]: dispatch 2024-03-26T02:15:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:45 smithi072 ceph-mon[41619]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:45 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-128"}]': finished 2024-03-26T02:15:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:45 smithi072 ceph-mon[41619]: osdmap e643: 8 total, 8 up, 8 in 2024-03-26T02:15:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:45 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-128", "mode": "writeback"}]: dispatch 2024-03-26T02:15:47.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:46 smithi173 ceph-mon[33839]: pgmap v992: 300 pgs: 300 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T02:15:47.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:46 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:15:47.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:46 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-128", "mode": "writeback"}]': finished 2024-03-26T02:15:47.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:46 smithi173 ceph-mon[33839]: osdmap e644: 8 total, 8 up, 8 in 2024-03-26T02:15:47.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:46 smithi072 ceph-mon[30177]: pgmap v992: 300 pgs: 300 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T02:15:47.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:46 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:15:47.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:46 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-128", "mode": "writeback"}]': finished 2024-03-26T02:15:47.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:46 smithi072 ceph-mon[30177]: osdmap e644: 8 total, 8 up, 8 in 2024-03-26T02:15:47.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:46 smithi072 ceph-mon[41619]: pgmap v992: 300 pgs: 300 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T02:15:47.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:46 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:15:47.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:46 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-128", "mode": "writeback"}]': finished 2024-03-26T02:15:47.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:46 smithi072 ceph-mon[41619]: osdmap e644: 8 total, 8 up, 8 in 2024-03-26T02:15:49.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:48 smithi173 ceph-mon[33839]: pgmap v995: 300 pgs: 300 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:15:49.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:48 smithi072 ceph-mon[30177]: pgmap v995: 300 pgs: 300 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:15:49.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:48 smithi072 ceph-mon[41619]: pgmap v995: 300 pgs: 300 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:15:51.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:50 smithi173 ceph-mon[33839]: pgmap v996: 300 pgs: 300 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-26T02:15:51.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.6", "id": [5, 3]}]: dispatch 2024-03-26T02:15:51.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.13", "id": [5, 3]}]: dispatch 2024-03-26T02:15:51.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.1a", "id": [2, 3]}]: dispatch 2024-03-26T02:15:51.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:50 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:51.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:50 smithi072 ceph-mon[30177]: pgmap v996: 300 pgs: 300 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-26T02:15:51.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.6", "id": [5, 3]}]: dispatch 2024-03-26T02:15:51.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.13", "id": [5, 3]}]: dispatch 2024-03-26T02:15:51.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.1a", "id": [2, 3]}]: dispatch 2024-03-26T02:15:51.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:50 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:51.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:50 smithi072 ceph-mon[41619]: pgmap v996: 300 pgs: 300 active+clean; 582 KiB data, 659 MiB used, 715 GiB / 715 GiB avail; 857 B/s rd, 0 op/s 2024-03-26T02:15:51.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.6", "id": [5, 3]}]: dispatch 2024-03-26T02:15:51.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.13", "id": [5, 3]}]: dispatch 2024-03-26T02:15:51.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.1a", "id": [2, 3]}]: dispatch 2024-03-26T02:15:51.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:50 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:15:51.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:15:51 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:15:52.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:51 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.6", "id": [5, 3]}]': finished 2024-03-26T02:15:52.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:51 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.13", "id": [5, 3]}]': finished 2024-03-26T02:15:52.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:51 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.1a", "id": [2, 3]}]': finished 2024-03-26T02:15:52.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:51 smithi173 ceph-mon[33839]: osdmap e645: 8 total, 8 up, 8 in 2024-03-26T02:15:52.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:51 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:15:52.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:51 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:15:52.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:51 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.6", "id": [5, 3]}]': finished 2024-03-26T02:15:52.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:51 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.13", "id": [5, 3]}]': finished 2024-03-26T02:15:52.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:51 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.1a", "id": [2, 3]}]': finished 2024-03-26T02:15:52.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:51 smithi072 ceph-mon[30177]: osdmap e645: 8 total, 8 up, 8 in 2024-03-26T02:15:52.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:51 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:15:52.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:51 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:15:52.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:51 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.6", "id": [5, 3]}]': finished 2024-03-26T02:15:52.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:51 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.13", "id": [5, 3]}]': finished 2024-03-26T02:15:52.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:51 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "321.1a", "id": [2, 3]}]': finished 2024-03-26T02:15:52.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:51 smithi072 ceph-mon[41619]: osdmap e645: 8 total, 8 up, 8 in 2024-03-26T02:15:52.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:51 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:15:52.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:51 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:15:53.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:52 smithi173 ceph-mon[33839]: pgmap v998: 300 pgs: 300 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-03-26T02:15:53.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:52 smithi173 ceph-mon[33839]: osdmap e646: 8 total, 8 up, 8 in 2024-03-26T02:15:53.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:52 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:15:53.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:15:53.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:15:53.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:15:53.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:15:53.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:15:53.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:52 smithi072 ceph-mon[30177]: pgmap v998: 300 pgs: 300 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-03-26T02:15:53.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:52 smithi072 ceph-mon[30177]: osdmap e646: 8 total, 8 up, 8 in 2024-03-26T02:15:53.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:52 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:15:53.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:15:53.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:15:53.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:15:53.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:15:53.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:15:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:52 smithi072 ceph-mon[41619]: pgmap v998: 300 pgs: 300 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-03-26T02:15:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:52 smithi072 ceph-mon[41619]: osdmap e646: 8 total, 8 up, 8 in 2024-03-26T02:15:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:52 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:15:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:15:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:15:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:15:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:15:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:15:54.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:53 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:15:54.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:53 smithi173 ceph-mon[33839]: osdmap e647: 8 total, 8 up, 8 in 2024-03-26T02:15:54.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:53 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-128"}]: dispatch 2024-03-26T02:15:54.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:53 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:15:54.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:53 smithi072 ceph-mon[30177]: osdmap e647: 8 total, 8 up, 8 in 2024-03-26T02:15:54.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:53 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-128"}]: dispatch 2024-03-26T02:15:54.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:53 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:15:54.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:53 smithi072 ceph-mon[41619]: osdmap e647: 8 total, 8 up, 8 in 2024-03-26T02:15:54.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:53 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-128"}]: dispatch 2024-03-26T02:15:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:55 smithi173 ceph-mon[33839]: pgmap v1001: 300 pgs: 300 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-03-26T02:15:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:55 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:15:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:55 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-128"}]': finished 2024-03-26T02:15:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:55 smithi173 ceph-mon[33839]: osdmap e648: 8 total, 8 up, 8 in 2024-03-26T02:15:55.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:15:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:15:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:15:55.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:55 smithi072 ceph-mon[30177]: pgmap v1001: 300 pgs: 300 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-03-26T02:15:55.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:55 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:15:55.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:55 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-128"}]': finished 2024-03-26T02:15:55.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:55 smithi072 ceph-mon[30177]: osdmap e648: 8 total, 8 up, 8 in 2024-03-26T02:15:55.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:55 smithi072 ceph-mon[41619]: pgmap v1001: 300 pgs: 300 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-03-26T02:15:55.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:55 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:15:55.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:55 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-128"}]': finished 2024-03-26T02:15:55.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:55 smithi072 ceph-mon[41619]: osdmap e648: 8 total, 8 up, 8 in 2024-03-26T02:15:56.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:56 smithi072 ceph-mon[30177]: osdmap e649: 8 total, 8 up, 8 in 2024-03-26T02:15:56.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:56 smithi072 ceph-mon[30177]: osdmap e650: 8 total, 8 up, 8 in 2024-03-26T02:15:56.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:56 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:15:56.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:56 smithi072 ceph-mon[41619]: osdmap e649: 8 total, 8 up, 8 in 2024-03-26T02:15:56.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:56 smithi072 ceph-mon[41619]: osdmap e650: 8 total, 8 up, 8 in 2024-03-26T02:15:56.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:56 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:15:56.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:56 smithi173 ceph-mon[33839]: osdmap e649: 8 total, 8 up, 8 in 2024-03-26T02:15:56.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:56 smithi173 ceph-mon[33839]: osdmap e650: 8 total, 8 up, 8 in 2024-03-26T02:15:56.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:56 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:15:57.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:57 smithi072 ceph-mon[30177]: pgmap v1004: 268 pgs: 268 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:15:57.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:57 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:15:57.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:57 smithi072 ceph-mon[30177]: osdmap e651: 8 total, 8 up, 8 in 2024-03-26T02:15:57.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:57 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:15:57.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:57 smithi072 ceph-mon[41619]: pgmap v1004: 268 pgs: 268 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:15:57.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:57 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:15:57.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:57 smithi072 ceph-mon[41619]: osdmap e651: 8 total, 8 up, 8 in 2024-03-26T02:15:57.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:57 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:15:57.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:57 smithi173 ceph-mon[33839]: pgmap v1004: 268 pgs: 268 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:15:57.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:57 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:15:57.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:57 smithi173 ceph-mon[33839]: osdmap e651: 8 total, 8 up, 8 in 2024-03-26T02:15:57.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:57 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:15:59.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:58 smithi072 ceph-mon[30177]: pgmap v1007: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:15:59.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:15:59.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:58 smithi072 ceph-mon[30177]: osdmap e652: 8 total, 8 up, 8 in 2024-03-26T02:15:59.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-130"}]: dispatch 2024-03-26T02:15:59.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:58 smithi072 ceph-mon[41619]: pgmap v1007: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:15:59.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:15:59.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:58 smithi072 ceph-mon[41619]: osdmap e652: 8 total, 8 up, 8 in 2024-03-26T02:15:59.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-130"}]: dispatch 2024-03-26T02:15:59.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:58 smithi173 ceph-mon[33839]: pgmap v1007: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:15:59.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:15:59.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:58 smithi173 ceph-mon[33839]: osdmap e652: 8 total, 8 up, 8 in 2024-03-26T02:15:59.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-130"}]: dispatch 2024-03-26T02:16:00.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:59 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-130"}]': finished 2024-03-26T02:16:00.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:59 smithi072 ceph-mon[30177]: osdmap e653: 8 total, 8 up, 8 in 2024-03-26T02:16:00.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:59 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-130", "mode": "writeback"}]: dispatch 2024-03-26T02:16:00.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:59 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:16:00.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:59 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-130", "mode": "writeback"}]': finished 2024-03-26T02:16:00.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:15:59 smithi072 ceph-mon[30177]: osdmap e654: 8 total, 8 up, 8 in 2024-03-26T02:16:00.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:59 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-130"}]': finished 2024-03-26T02:16:00.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:59 smithi072 ceph-mon[41619]: osdmap e653: 8 total, 8 up, 8 in 2024-03-26T02:16:00.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:59 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-130", "mode": "writeback"}]: dispatch 2024-03-26T02:16:00.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:59 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:16:00.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:59 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-130", "mode": "writeback"}]': finished 2024-03-26T02:16:00.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:15:59 smithi072 ceph-mon[41619]: osdmap e654: 8 total, 8 up, 8 in 2024-03-26T02:16:00.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:59 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-130"}]': finished 2024-03-26T02:16:00.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:59 smithi173 ceph-mon[33839]: osdmap e653: 8 total, 8 up, 8 in 2024-03-26T02:16:00.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:59 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-130", "mode": "writeback"}]: dispatch 2024-03-26T02:16:00.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:59 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:16:00.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:59 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-130", "mode": "writeback"}]': finished 2024-03-26T02:16:00.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:15:59 smithi173 ceph-mon[33839]: osdmap e654: 8 total, 8 up, 8 in 2024-03-26T02:16:01.213 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:00 smithi173 ceph-mon[33839]: pgmap v1010: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:16:01.213 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:16:01.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:00 smithi072 ceph-mon[30177]: pgmap v1010: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:16:01.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:16:01.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:00 smithi072 ceph-mon[41619]: pgmap v1010: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 660 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:16:01.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:16:01.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:16:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:16:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:16:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:01 smithi072 ceph-mon[30177]: osdmap e655: 8 total, 8 up, 8 in 2024-03-26T02:16:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-130"}]: dispatch 2024-03-26T02:16:02.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:01 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:16:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:16:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:01 smithi072 ceph-mon[41619]: osdmap e655: 8 total, 8 up, 8 in 2024-03-26T02:16:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-130"}]: dispatch 2024-03-26T02:16:02.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:01 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:16:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:16:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:01 smithi173 ceph-mon[33839]: osdmap e655: 8 total, 8 up, 8 in 2024-03-26T02:16:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-130"}]: dispatch 2024-03-26T02:16:02.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:01 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:16:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:03 smithi072 ceph-mon[30177]: pgmap v1013: 300 pgs: 300 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-26T02:16:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:03 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:16:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-130"}]': finished 2024-03-26T02:16:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:03 smithi072 ceph-mon[30177]: osdmap e656: 8 total, 8 up, 8 in 2024-03-26T02:16:03.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:03 smithi072 ceph-mon[41619]: pgmap v1013: 300 pgs: 300 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-26T02:16:03.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:03 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:16:03.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-130"}]': finished 2024-03-26T02:16:03.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:03 smithi072 ceph-mon[41619]: osdmap e656: 8 total, 8 up, 8 in 2024-03-26T02:16:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:03 smithi173 ceph-mon[33839]: pgmap v1013: 300 pgs: 300 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-26T02:16:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:03 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:16:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-130"}]': finished 2024-03-26T02:16:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:03 smithi173 ceph-mon[33839]: osdmap e656: 8 total, 8 up, 8 in 2024-03-26T02:16:04.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:04 smithi072 ceph-mon[30177]: osdmap e657: 8 total, 8 up, 8 in 2024-03-26T02:16:04.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:04 smithi072 ceph-mon[30177]: osdmap e658: 8 total, 8 up, 8 in 2024-03-26T02:16:04.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:04 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:16:04.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:04 smithi072 ceph-mon[41619]: osdmap e657: 8 total, 8 up, 8 in 2024-03-26T02:16:04.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:04 smithi072 ceph-mon[41619]: osdmap e658: 8 total, 8 up, 8 in 2024-03-26T02:16:04.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:04 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:16:04.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:04 smithi173 ceph-mon[33839]: osdmap e657: 8 total, 8 up, 8 in 2024-03-26T02:16:04.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:04 smithi173 ceph-mon[33839]: osdmap e658: 8 total, 8 up, 8 in 2024-03-26T02:16:04.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:04 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:16:05.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:16:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:16:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:16:05.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:05 smithi072 ceph-mon[30177]: pgmap v1016: 268 pgs: 268 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:16:05.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:16:05.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:05 smithi072 ceph-mon[30177]: osdmap e659: 8 total, 8 up, 8 in 2024-03-26T02:16:05.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:16:05.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:05 smithi072 ceph-mon[41619]: pgmap v1016: 268 pgs: 268 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:16:05.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:16:05.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:05 smithi072 ceph-mon[41619]: osdmap e659: 8 total, 8 up, 8 in 2024-03-26T02:16:05.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:16:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:05 smithi173 ceph-mon[33839]: pgmap v1016: 268 pgs: 268 active+clean; 582 KiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T02:16:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:16:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:05 smithi173 ceph-mon[33839]: osdmap e659: 8 total, 8 up, 8 in 2024-03-26T02:16:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:16:07.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:06 smithi072 ceph-mon[30177]: pgmap v1019: 300 pgs: 300 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:16:07.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:16:07.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:06 smithi072 ceph-mon[30177]: osdmap e660: 8 total, 8 up, 8 in 2024-03-26T02:16:07.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-132"}]: dispatch 2024-03-26T02:16:07.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:06 smithi072 ceph-mon[41619]: pgmap v1019: 300 pgs: 300 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:16:07.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:16:07.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:06 smithi072 ceph-mon[41619]: osdmap e660: 8 total, 8 up, 8 in 2024-03-26T02:16:07.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-132"}]: dispatch 2024-03-26T02:16:07.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:06 smithi173 ceph-mon[33839]: pgmap v1019: 300 pgs: 300 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:16:07.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:16:07.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:06 smithi173 ceph-mon[33839]: osdmap e660: 8 total, 8 up, 8 in 2024-03-26T02:16:07.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-132"}]: dispatch 2024-03-26T02:16:08.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-132"}]': finished 2024-03-26T02:16:08.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:07 smithi072 ceph-mon[30177]: osdmap e661: 8 total, 8 up, 8 in 2024-03-26T02:16:08.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-132", "mode": "writeback"}]: dispatch 2024-03-26T02:16:08.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:07 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:16:08.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:07 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:16:08.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:07 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:16:08.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-132", "mode": "writeback"}]': finished 2024-03-26T02:16:08.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-132"}]': finished 2024-03-26T02:16:08.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:07 smithi072 ceph-mon[41619]: osdmap e661: 8 total, 8 up, 8 in 2024-03-26T02:16:08.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-132", "mode": "writeback"}]: dispatch 2024-03-26T02:16:08.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:07 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:16:08.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:07 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:16:08.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:07 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:16:08.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-132", "mode": "writeback"}]': finished 2024-03-26T02:16:08.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-132"}]': finished 2024-03-26T02:16:08.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:07 smithi173 ceph-mon[33839]: osdmap e661: 8 total, 8 up, 8 in 2024-03-26T02:16:08.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-132", "mode": "writeback"}]: dispatch 2024-03-26T02:16:08.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:07 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:16:08.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:07 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:16:08.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:07 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:16:08.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-132", "mode": "writeback"}]': finished 2024-03-26T02:16:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:08 smithi072 ceph-mon[30177]: pgmap v1022: 300 pgs: 300 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:08 smithi072 ceph-mon[30177]: osdmap e662: 8 total, 8 up, 8 in 2024-03-26T02:16:09.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:08 smithi072 ceph-mon[41619]: pgmap v1022: 300 pgs: 300 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:09.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:08 smithi072 ceph-mon[41619]: osdmap e662: 8 total, 8 up, 8 in 2024-03-26T02:16:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:08 smithi173 ceph-mon[33839]: pgmap v1022: 300 pgs: 300 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:08 smithi173 ceph-mon[33839]: osdmap e662: 8 total, 8 up, 8 in 2024-03-26T02:16:10.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:09 smithi072 ceph-mon[30177]: osdmap e663: 8 total, 8 up, 8 in 2024-03-26T02:16:10.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:09 smithi072 ceph-mon[41619]: osdmap e663: 8 total, 8 up, 8 in 2024-03-26T02:16:10.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:09 smithi173 ceph-mon[33839]: osdmap e663: 8 total, 8 up, 8 in 2024-03-26T02:16:11.214 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:10 smithi173 ceph-mon[33839]: pgmap v1025: 300 pgs: 300 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:16:11.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:10 smithi173 ceph-mon[33839]: osdmap e664: 8 total, 8 up, 8 in 2024-03-26T02:16:11.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:10 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:16:11.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:10 smithi072 ceph-mon[30177]: pgmap v1025: 300 pgs: 300 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:16:11.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:10 smithi072 ceph-mon[30177]: osdmap e664: 8 total, 8 up, 8 in 2024-03-26T02:16:11.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:10 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:16:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:10 smithi072 ceph-mon[41619]: pgmap v1025: 300 pgs: 300 active+clean; 582 KiB data, 691 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:16:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:10 smithi072 ceph-mon[41619]: osdmap e664: 8 total, 8 up, 8 in 2024-03-26T02:16:11.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:10 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:16:11.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:16:11 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:16:12.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:11 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:16:12.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:11 smithi072 ceph-mon[30177]: osdmap e665: 8 total, 8 up, 8 in 2024-03-26T02:16:12.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:11 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-132"}]: dispatch 2024-03-26T02:16:12.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:11 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:16:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:11 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:16:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:11 smithi072 ceph-mon[41619]: osdmap e665: 8 total, 8 up, 8 in 2024-03-26T02:16:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:11 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-132"}]: dispatch 2024-03-26T02:16:12.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:11 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:16:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:11 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:16:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:11 smithi173 ceph-mon[33839]: osdmap e665: 8 total, 8 up, 8 in 2024-03-26T02:16:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:11 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-132"}]: dispatch 2024-03-26T02:16:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:11 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:16:13.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:12 smithi072 ceph-mon[30177]: pgmap v1028: 300 pgs: 300 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T02:16:13.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-132"}]': finished 2024-03-26T02:16:13.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:12 smithi072 ceph-mon[30177]: osdmap e666: 8 total, 8 up, 8 in 2024-03-26T02:16:13.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:12 smithi072 ceph-mon[41619]: pgmap v1028: 300 pgs: 300 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T02:16:13.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-132"}]': finished 2024-03-26T02:16:13.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:12 smithi072 ceph-mon[41619]: osdmap e666: 8 total, 8 up, 8 in 2024-03-26T02:16:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:12 smithi173 ceph-mon[33839]: pgmap v1028: 300 pgs: 300 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T02:16:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-132"}]': finished 2024-03-26T02:16:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:12 smithi173 ceph-mon[33839]: osdmap e666: 8 total, 8 up, 8 in 2024-03-26T02:16:14.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:13 smithi072 ceph-mon[30177]: osdmap e667: 8 total, 8 up, 8 in 2024-03-26T02:16:14.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:13 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:16:14.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:13 smithi072 ceph-mon[41619]: osdmap e667: 8 total, 8 up, 8 in 2024-03-26T02:16:14.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:13 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:16:14.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:13 smithi173 ceph-mon[33839]: osdmap e667: 8 total, 8 up, 8 in 2024-03-26T02:16:14.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:13 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:16:15.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:16:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:16:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:16:15.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:14 smithi072 ceph-mon[30177]: pgmap v1031: 300 pgs: 300 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T02:16:15.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:14 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:16:15.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:14 smithi072 ceph-mon[30177]: osdmap e668: 8 total, 8 up, 8 in 2024-03-26T02:16:15.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:14 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-132"}]: dispatch 2024-03-26T02:16:15.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:14 smithi072 ceph-mon[41619]: pgmap v1031: 300 pgs: 300 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T02:16:15.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:14 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:16:15.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:14 smithi072 ceph-mon[41619]: osdmap e668: 8 total, 8 up, 8 in 2024-03-26T02:16:15.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:14 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-132"}]: dispatch 2024-03-26T02:16:15.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:14 smithi173 ceph-mon[33839]: pgmap v1031: 300 pgs: 300 active+clean; 582 KiB data, 692 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T02:16:15.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:14 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:16:15.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:14 smithi173 ceph-mon[33839]: osdmap e668: 8 total, 8 up, 8 in 2024-03-26T02:16:15.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:14 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-132"}]: dispatch 2024-03-26T02:16:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:15 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:16:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-132"}]': finished 2024-03-26T02:16:16.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:15 smithi072 ceph-mon[30177]: osdmap e669: 8 total, 8 up, 8 in 2024-03-26T02:16:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:15 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:16:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-132"}]': finished 2024-03-26T02:16:16.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:15 smithi072 ceph-mon[41619]: osdmap e669: 8 total, 8 up, 8 in 2024-03-26T02:16:16.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:15 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:16:16.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-132"}]': finished 2024-03-26T02:16:16.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:15 smithi173 ceph-mon[33839]: osdmap e669: 8 total, 8 up, 8 in 2024-03-26T02:16:17.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:16 smithi072 ceph-mon[41619]: pgmap v1034: 300 pgs: 300 active+clean; 582 KiB data, 693 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T02:16:17.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:16 smithi072 ceph-mon[41619]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:16:17.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:16 smithi072 ceph-mon[41619]: osdmap e670: 8 total, 8 up, 8 in 2024-03-26T02:16:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:16 smithi072 ceph-mon[30177]: pgmap v1034: 300 pgs: 300 active+clean; 582 KiB data, 693 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T02:16:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:16 smithi072 ceph-mon[30177]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:16:17.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:16 smithi072 ceph-mon[30177]: osdmap e670: 8 total, 8 up, 8 in 2024-03-26T02:16:17.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:16 smithi173 ceph-mon[33839]: pgmap v1034: 300 pgs: 300 active+clean; 582 KiB data, 693 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T02:16:17.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:16 smithi173 ceph-mon[33839]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:16:17.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:16 smithi173 ceph-mon[33839]: osdmap e670: 8 total, 8 up, 8 in 2024-03-26T02:16:18.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:17 smithi072 ceph-mon[30177]: osdmap e671: 8 total, 8 up, 8 in 2024-03-26T02:16:18.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:17 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:16:18.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:17 smithi072 ceph-mon[41619]: osdmap e671: 8 total, 8 up, 8 in 2024-03-26T02:16:18.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:17 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:16:18.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:17 smithi173 ceph-mon[33839]: osdmap e671: 8 total, 8 up, 8 in 2024-03-26T02:16:18.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:17 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:16:19.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:18 smithi072 ceph-mon[30177]: pgmap v1037: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 693 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:19.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:16:19.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:18 smithi072 ceph-mon[30177]: osdmap e672: 8 total, 8 up, 8 in 2024-03-26T02:16:19.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:16:19.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:18 smithi072 ceph-mon[41619]: pgmap v1037: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 693 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:19.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:16:19.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:18 smithi072 ceph-mon[41619]: osdmap e672: 8 total, 8 up, 8 in 2024-03-26T02:16:19.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:16:19.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:18 smithi173 ceph-mon[33839]: pgmap v1037: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 693 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:19.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:16:19.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:18 smithi173 ceph-mon[33839]: osdmap e672: 8 total, 8 up, 8 in 2024-03-26T02:16:19.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:16:20.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:16:20.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:19 smithi072 ceph-mon[30177]: osdmap e673: 8 total, 8 up, 8 in 2024-03-26T02:16:20.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-134"}]: dispatch 2024-03-26T02:16:20.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:16:20.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:19 smithi072 ceph-mon[41619]: osdmap e673: 8 total, 8 up, 8 in 2024-03-26T02:16:20.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-134"}]: dispatch 2024-03-26T02:16:20.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:16:20.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:19 smithi173 ceph-mon[33839]: osdmap e673: 8 total, 8 up, 8 in 2024-03-26T02:16:20.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-134"}]: dispatch 2024-03-26T02:16:21.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:20 smithi072 ceph-mon[30177]: pgmap v1040: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 693 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:16:21.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-134"}]': finished 2024-03-26T02:16:21.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:20 smithi072 ceph-mon[30177]: osdmap e674: 8 total, 8 up, 8 in 2024-03-26T02:16:21.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-134", "mode": "writeback"}]: dispatch 2024-03-26T02:16:21.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:20 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:16:21.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-134", "mode": "writeback"}]': finished 2024-03-26T02:16:21.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:20 smithi072 ceph-mon[30177]: osdmap e675: 8 total, 8 up, 8 in 2024-03-26T02:16:21.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:16:21.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:20 smithi072 ceph-mon[41619]: pgmap v1040: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 693 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:16:21.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-134"}]': finished 2024-03-26T02:16:21.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:20 smithi072 ceph-mon[41619]: osdmap e674: 8 total, 8 up, 8 in 2024-03-26T02:16:21.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-134", "mode": "writeback"}]: dispatch 2024-03-26T02:16:21.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:20 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:16:21.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-134", "mode": "writeback"}]': finished 2024-03-26T02:16:21.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:20 smithi072 ceph-mon[41619]: osdmap e675: 8 total, 8 up, 8 in 2024-03-26T02:16:21.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:16:21.233 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:20 smithi173 ceph-mon[33839]: pgmap v1040: 300 pgs: 32 unknown, 268 active+clean; 582 KiB data, 693 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:16:21.233 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-134"}]': finished 2024-03-26T02:16:21.233 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:20 smithi173 ceph-mon[33839]: osdmap e674: 8 total, 8 up, 8 in 2024-03-26T02:16:21.233 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-134", "mode": "writeback"}]: dispatch 2024-03-26T02:16:21.233 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:20 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:16:21.233 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-134", "mode": "writeback"}]': finished 2024-03-26T02:16:21.233 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:20 smithi173 ceph-mon[33839]: osdmap e675: 8 total, 8 up, 8 in 2024-03-26T02:16:21.233 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:16:21.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:16:21 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:16:22.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:21 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:16:22.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:16:22.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:21 smithi173 ceph-mon[33839]: osdmap e676: 8 total, 8 up, 8 in 2024-03-26T02:16:22.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-134"}]: dispatch 2024-03-26T02:16:22.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:21 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:16:22.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:16:22.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:21 smithi072 ceph-mon[30177]: osdmap e676: 8 total, 8 up, 8 in 2024-03-26T02:16:22.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-134"}]: dispatch 2024-03-26T02:16:22.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:21 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:16:22.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:16:22.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:21 smithi072 ceph-mon[41619]: osdmap e676: 8 total, 8 up, 8 in 2024-03-26T02:16:22.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-134"}]: dispatch 2024-03-26T02:16:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:23 smithi173 ceph-mon[33839]: pgmap v1043: 300 pgs: 300 active+clean; 582 KiB data, 693 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:23 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:16:23.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:23 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:16:23.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:23 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:16:23.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:23 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-134"}]': finished 2024-03-26T02:16:23.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:23 smithi173 ceph-mon[33839]: osdmap e677: 8 total, 8 up, 8 in 2024-03-26T02:16:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:23 smithi072 ceph-mon[30177]: pgmap v1043: 300 pgs: 300 active+clean; 582 KiB data, 693 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:23 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:16:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:23 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:16:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:23 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:16:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:23 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-134"}]': finished 2024-03-26T02:16:23.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:23 smithi072 ceph-mon[30177]: osdmap e677: 8 total, 8 up, 8 in 2024-03-26T02:16:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:23 smithi072 ceph-mon[41619]: pgmap v1043: 300 pgs: 300 active+clean; 582 KiB data, 693 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:23 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:16:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:23 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:16:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:23 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:16:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:23 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-134"}]': finished 2024-03-26T02:16:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:23 smithi072 ceph-mon[41619]: osdmap e677: 8 total, 8 up, 8 in 2024-03-26T02:16:25.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:24 smithi173 ceph-mon[33839]: pgmap v1046: 300 pgs: 300 active+clean; 582 KiB data, 693 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:25.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:24 smithi173 ceph-mon[33839]: osdmap e678: 8 total, 8 up, 8 in 2024-03-26T02:16:25.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:24 smithi173 ceph-mon[33839]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:16:25.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:24 smithi072 ceph-mon[41619]: pgmap v1046: 300 pgs: 300 active+clean; 582 KiB data, 693 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:25.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:24 smithi072 ceph-mon[41619]: osdmap e678: 8 total, 8 up, 8 in 2024-03-26T02:16:25.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:24 smithi072 ceph-mon[41619]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:16:25.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:24 smithi072 ceph-mon[30177]: pgmap v1046: 300 pgs: 300 active+clean; 582 KiB data, 693 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:25.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:24 smithi072 ceph-mon[30177]: osdmap e678: 8 total, 8 up, 8 in 2024-03-26T02:16:25.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:24 smithi072 ceph-mon[30177]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:16:25.220 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:16:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:16:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:16:26.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:25 smithi173 ceph-mon[33839]: osdmap e679: 8 total, 8 up, 8 in 2024-03-26T02:16:26.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:16:26.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:25 smithi072 ceph-mon[30177]: osdmap e679: 8 total, 8 up, 8 in 2024-03-26T02:16:26.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:16:26.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:25 smithi072 ceph-mon[41619]: osdmap e679: 8 total, 8 up, 8 in 2024-03-26T02:16:26.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:16:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:26 smithi173 ceph-mon[33839]: pgmap v1049: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 694 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T02:16:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:26 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:16:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:26 smithi173 ceph-mon[33839]: osdmap e680: 8 total, 8 up, 8 in 2024-03-26T02:16:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:26 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:16:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:26 smithi072 ceph-mon[30177]: pgmap v1049: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 694 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T02:16:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:26 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:16:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:26 smithi072 ceph-mon[30177]: osdmap e680: 8 total, 8 up, 8 in 2024-03-26T02:16:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:26 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:16:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:26 smithi072 ceph-mon[41619]: pgmap v1049: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 694 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T02:16:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:26 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:16:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:26 smithi072 ceph-mon[41619]: osdmap e680: 8 total, 8 up, 8 in 2024-03-26T02:16:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:26 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:16:28.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:27 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:16:28.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:27 smithi173 ceph-mon[33839]: osdmap e681: 8 total, 8 up, 8 in 2024-03-26T02:16:28.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:27 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-136"}]: dispatch 2024-03-26T02:16:28.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:27 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:16:28.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:27 smithi072 ceph-mon[30177]: osdmap e681: 8 total, 8 up, 8 in 2024-03-26T02:16:28.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:27 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-136"}]: dispatch 2024-03-26T02:16:28.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:27 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:16:28.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:27 smithi072 ceph-mon[41619]: osdmap e681: 8 total, 8 up, 8 in 2024-03-26T02:16:28.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:27 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-136"}]: dispatch 2024-03-26T02:16:29.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:28 smithi173 ceph-mon[33839]: pgmap v1052: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 694 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T02:16:29.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:28 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-136"}]': finished 2024-03-26T02:16:29.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:28 smithi173 ceph-mon[33839]: osdmap e682: 8 total, 8 up, 8 in 2024-03-26T02:16:29.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:28 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-136", "mode": "writeback"}]: dispatch 2024-03-26T02:16:29.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:28 smithi072 ceph-mon[30177]: pgmap v1052: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 694 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T02:16:29.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:28 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-136"}]': finished 2024-03-26T02:16:29.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:28 smithi072 ceph-mon[30177]: osdmap e682: 8 total, 8 up, 8 in 2024-03-26T02:16:29.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:28 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-136", "mode": "writeback"}]: dispatch 2024-03-26T02:16:29.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:28 smithi072 ceph-mon[41619]: pgmap v1052: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 694 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T02:16:29.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:28 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-136"}]': finished 2024-03-26T02:16:29.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:28 smithi072 ceph-mon[41619]: osdmap e682: 8 total, 8 up, 8 in 2024-03-26T02:16:29.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:28 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-136", "mode": "writeback"}]: dispatch 2024-03-26T02:16:30.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:29 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:16:30.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:29 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-136", "mode": "writeback"}]': finished 2024-03-26T02:16:30.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:29 smithi173 ceph-mon[33839]: osdmap e683: 8 total, 8 up, 8 in 2024-03-26T02:16:30.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:29 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:16:30.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:29 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:16:30.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:29 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-136", "mode": "writeback"}]': finished 2024-03-26T02:16:30.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:29 smithi072 ceph-mon[30177]: osdmap e683: 8 total, 8 up, 8 in 2024-03-26T02:16:30.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:29 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:16:30.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:29 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:16:30.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:29 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-136", "mode": "writeback"}]': finished 2024-03-26T02:16:30.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:29 smithi072 ceph-mon[41619]: osdmap e683: 8 total, 8 up, 8 in 2024-03-26T02:16:30.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:29 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:16:31.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:30 smithi173 ceph-mon[33839]: pgmap v1055: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 694 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:16:31.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:16:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:30 smithi173 ceph-mon[33839]: osdmap e684: 8 total, 8 up, 8 in 2024-03-26T02:16:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-136"}]: dispatch 2024-03-26T02:16:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:30 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:16:31.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:30 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:16:31.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:30 smithi072 ceph-mon[30177]: pgmap v1055: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 694 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:16:31.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:16:31.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:30 smithi072 ceph-mon[30177]: osdmap e684: 8 total, 8 up, 8 in 2024-03-26T02:16:31.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-136"}]: dispatch 2024-03-26T02:16:31.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:30 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:16:31.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:30 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:16:31.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:30 smithi072 ceph-mon[41619]: pgmap v1055: 300 pgs: 32 creating+peering, 268 active+clean; 582 KiB data, 694 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:16:31.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:16:31.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:30 smithi072 ceph-mon[41619]: osdmap e684: 8 total, 8 up, 8 in 2024-03-26T02:16:31.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-136"}]: dispatch 2024-03-26T02:16:31.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:30 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:16:31.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:30 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:16:31.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:16:31 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:16:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:31 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-136"}]': finished 2024-03-26T02:16:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:31 smithi173 ceph-mon[33839]: osdmap e685: 8 total, 8 up, 8 in 2024-03-26T02:16:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:31 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:16:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:31 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-136"}]': finished 2024-03-26T02:16:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:31 smithi072 ceph-mon[30177]: osdmap e685: 8 total, 8 up, 8 in 2024-03-26T02:16:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:31 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:16:32.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:31 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-136"}]': finished 2024-03-26T02:16:32.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:31 smithi072 ceph-mon[41619]: osdmap e685: 8 total, 8 up, 8 in 2024-03-26T02:16:32.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:31 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:16:33.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:32 smithi072 ceph-mon[30177]: pgmap v1058: 300 pgs: 300 active+clean; 4.4 MiB data, 700 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-26T02:16:33.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:32 smithi072 ceph-mon[30177]: osdmap e686: 8 total, 8 up, 8 in 2024-03-26T02:16:33.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:32 smithi072 ceph-mon[41619]: pgmap v1058: 300 pgs: 300 active+clean; 4.4 MiB data, 700 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-26T02:16:33.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:32 smithi072 ceph-mon[41619]: osdmap e686: 8 total, 8 up, 8 in 2024-03-26T02:16:33.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:32 smithi173 ceph-mon[33839]: pgmap v1058: 300 pgs: 300 active+clean; 4.4 MiB data, 700 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-26T02:16:33.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:32 smithi173 ceph-mon[33839]: osdmap e686: 8 total, 8 up, 8 in 2024-03-26T02:16:34.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:33 smithi072 ceph-mon[30177]: osdmap e687: 8 total, 8 up, 8 in 2024-03-26T02:16:34.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:33 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:16:34.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:33 smithi072 ceph-mon[41619]: osdmap e687: 8 total, 8 up, 8 in 2024-03-26T02:16:34.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:33 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:16:34.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:33 smithi173 ceph-mon[33839]: osdmap e687: 8 total, 8 up, 8 in 2024-03-26T02:16:34.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:33 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:16:35.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:16:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:16:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:16:35.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:34 smithi072 ceph-mon[30177]: pgmap v1061: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 700 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-26T02:16:35.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:34 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:16:35.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:34 smithi072 ceph-mon[30177]: osdmap e688: 8 total, 8 up, 8 in 2024-03-26T02:16:35.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:34 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:16:35.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:34 smithi072 ceph-mon[41619]: pgmap v1061: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 700 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-26T02:16:35.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:34 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:16:35.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:34 smithi072 ceph-mon[41619]: osdmap e688: 8 total, 8 up, 8 in 2024-03-26T02:16:35.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:34 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:16:35.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:34 smithi173 ceph-mon[33839]: pgmap v1061: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 700 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-26T02:16:35.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:34 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:16:35.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:34 smithi173 ceph-mon[33839]: osdmap e688: 8 total, 8 up, 8 in 2024-03-26T02:16:35.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:34 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:16:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:35 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:16:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:35 smithi072 ceph-mon[30177]: osdmap e689: 8 total, 8 up, 8 in 2024-03-26T02:16:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:35 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:16:36.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:35 smithi072 ceph-mon[30177]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:16:36.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:35 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:16:36.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:35 smithi072 ceph-mon[41619]: osdmap e689: 8 total, 8 up, 8 in 2024-03-26T02:16:36.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:35 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:16:36.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:35 smithi072 ceph-mon[41619]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:16:36.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:35 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:16:36.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:35 smithi173 ceph-mon[33839]: osdmap e689: 8 total, 8 up, 8 in 2024-03-26T02:16:36.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:35 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:16:36.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:35 smithi173 ceph-mon[33839]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:16:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:36 smithi072 ceph-mon[30177]: pgmap v1064: 300 pgs: 300 active+clean; 4.4 MiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:37.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:36 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-138","var": "hit_set_count","val": "2"}]': finished 2024-03-26T02:16:37.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:36 smithi072 ceph-mon[30177]: osdmap e690: 8 total, 8 up, 8 in 2024-03-26T02:16:37.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:36 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:16:37.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:36 smithi072 ceph-mon[41619]: pgmap v1064: 300 pgs: 300 active+clean; 4.4 MiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:37.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:36 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-138","var": "hit_set_count","val": "2"}]': finished 2024-03-26T02:16:37.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:36 smithi072 ceph-mon[41619]: osdmap e690: 8 total, 8 up, 8 in 2024-03-26T02:16:37.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:36 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:16:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:36 smithi173 ceph-mon[33839]: pgmap v1064: 300 pgs: 300 active+clean; 4.4 MiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:36 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-138","var": "hit_set_count","val": "2"}]': finished 2024-03-26T02:16:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:36 smithi173 ceph-mon[33839]: osdmap e690: 8 total, 8 up, 8 in 2024-03-26T02:16:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:36 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:16:38.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:37 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-138","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:16:38.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:37 smithi072 ceph-mon[30177]: osdmap e691: 8 total, 8 up, 8 in 2024-03-26T02:16:38.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:37 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T02:16:38.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:37 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:16:38.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:37 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:16:38.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:37 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T02:16:38.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:37 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-138","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:16:38.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:37 smithi072 ceph-mon[41619]: osdmap e691: 8 total, 8 up, 8 in 2024-03-26T02:16:38.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:37 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T02:16:38.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:37 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:16:38.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:37 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:16:38.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:37 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T02:16:38.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:37 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-138","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:16:38.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:37 smithi173 ceph-mon[33839]: osdmap e691: 8 total, 8 up, 8 in 2024-03-26T02:16:38.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:37 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T02:16:38.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:37 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:16:38.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:37 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:16:38.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:37 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T02:16:39.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:38 smithi072 ceph-mon[30177]: pgmap v1067: 300 pgs: 300 active+clean; 4.4 MiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:39.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:38 smithi072 ceph-mon[30177]: osdmap e692: 8 total, 8 up, 8 in 2024-03-26T02:16:39.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:38 smithi072 ceph-mon[41619]: pgmap v1067: 300 pgs: 300 active+clean; 4.4 MiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:39.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:38 smithi072 ceph-mon[41619]: osdmap e692: 8 total, 8 up, 8 in 2024-03-26T02:16:39.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:38 smithi173 ceph-mon[33839]: pgmap v1067: 300 pgs: 300 active+clean; 4.4 MiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:39.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:38 smithi173 ceph-mon[33839]: osdmap e692: 8 total, 8 up, 8 in 2024-03-26T02:16:40.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:16:40.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-138"}]: dispatch 2024-03-26T02:16:40.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:16:40.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-138"}]: dispatch 2024-03-26T02:16:40.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:16:40.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-138"}]: dispatch 2024-03-26T02:16:41.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:41 smithi072 ceph-mon[30177]: pgmap v1069: 300 pgs: 300 active+clean; 4.4 MiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:16:41.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:41 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-138"}]': finished 2024-03-26T02:16:41.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:41 smithi072 ceph-mon[30177]: osdmap e693: 8 total, 8 up, 8 in 2024-03-26T02:16:41.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:41 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:16:41.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:41 smithi072 ceph-mon[41619]: pgmap v1069: 300 pgs: 300 active+clean; 4.4 MiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:16:41.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:41 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-138"}]': finished 2024-03-26T02:16:41.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:41 smithi072 ceph-mon[41619]: osdmap e693: 8 total, 8 up, 8 in 2024-03-26T02:16:41.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:41 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:16:41.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:16:41 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:16:41.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:41 smithi173 ceph-mon[33839]: pgmap v1069: 300 pgs: 300 active+clean; 4.4 MiB data, 705 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:16:41.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:41 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-138"}]': finished 2024-03-26T02:16:41.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:41 smithi173 ceph-mon[33839]: osdmap e693: 8 total, 8 up, 8 in 2024-03-26T02:16:41.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:41 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:16:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:42 smithi072 ceph-mon[30177]: osdmap e694: 8 total, 8 up, 8 in 2024-03-26T02:16:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:42 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:16:42.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:42 smithi072 ceph-mon[30177]: osdmap e695: 8 total, 8 up, 8 in 2024-03-26T02:16:42.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:42 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:16:42.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:42 smithi072 ceph-mon[41619]: osdmap e694: 8 total, 8 up, 8 in 2024-03-26T02:16:42.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:42 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:16:42.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:42 smithi072 ceph-mon[41619]: osdmap e695: 8 total, 8 up, 8 in 2024-03-26T02:16:42.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:42 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:16:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:42 smithi173 ceph-mon[33839]: osdmap e694: 8 total, 8 up, 8 in 2024-03-26T02:16:42.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:42 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:16:42.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:42 smithi173 ceph-mon[33839]: osdmap e695: 8 total, 8 up, 8 in 2024-03-26T02:16:42.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:42 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:16:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:43 smithi072 ceph-mon[30177]: pgmap v1072: 268 pgs: 268 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:16:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:43 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:16:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:43 smithi072 ceph-mon[30177]: osdmap e696: 8 total, 8 up, 8 in 2024-03-26T02:16:43.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:43 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:16:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:43 smithi072 ceph-mon[41619]: pgmap v1072: 268 pgs: 268 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:16:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:43 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:16:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:43 smithi072 ceph-mon[41619]: osdmap e696: 8 total, 8 up, 8 in 2024-03-26T02:16:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:43 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:16:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:43 smithi173 ceph-mon[33839]: pgmap v1072: 268 pgs: 268 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:16:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:43 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:16:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:43 smithi173 ceph-mon[33839]: osdmap e696: 8 total, 8 up, 8 in 2024-03-26T02:16:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:43 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:16:45.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:16:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:16:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:16:45.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:44 smithi072 ceph-mon[30177]: pgmap v1075: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:45.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:44 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:16:45.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:44 smithi072 ceph-mon[30177]: osdmap e697: 8 total, 8 up, 8 in 2024-03-26T02:16:45.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:44 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T02:16:45.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:44 smithi072 ceph-mon[41619]: pgmap v1075: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:45.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:44 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:16:45.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:44 smithi072 ceph-mon[41619]: osdmap e697: 8 total, 8 up, 8 in 2024-03-26T02:16:45.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:44 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T02:16:45.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:44 smithi173 ceph-mon[33839]: pgmap v1075: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:45.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:44 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:16:45.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:44 smithi173 ceph-mon[33839]: osdmap e697: 8 total, 8 up, 8 in 2024-03-26T02:16:45.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:44 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T02:16:46.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:45 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-140","var": "hit_set_count","val": "3"}]': finished 2024-03-26T02:16:46.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:45 smithi072 ceph-mon[30177]: osdmap e698: 8 total, 8 up, 8 in 2024-03-26T02:16:46.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:45 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T02:16:46.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:45 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-140","var": "hit_set_period","val": "3"}]': finished 2024-03-26T02:16:46.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:45 smithi072 ceph-mon[30177]: osdmap e699: 8 total, 8 up, 8 in 2024-03-26T02:16:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:45 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-140","var": "hit_set_count","val": "3"}]': finished 2024-03-26T02:16:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:45 smithi072 ceph-mon[41619]: osdmap e698: 8 total, 8 up, 8 in 2024-03-26T02:16:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:45 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T02:16:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:45 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-140","var": "hit_set_period","val": "3"}]': finished 2024-03-26T02:16:46.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:45 smithi072 ceph-mon[41619]: osdmap e699: 8 total, 8 up, 8 in 2024-03-26T02:16:46.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:45 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-140","var": "hit_set_count","val": "3"}]': finished 2024-03-26T02:16:46.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:45 smithi173 ceph-mon[33839]: osdmap e698: 8 total, 8 up, 8 in 2024-03-26T02:16:46.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:45 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T02:16:46.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:45 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-140","var": "hit_set_period","val": "3"}]': finished 2024-03-26T02:16:46.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:45 smithi173 ceph-mon[33839]: osdmap e699: 8 total, 8 up, 8 in 2024-03-26T02:16:47.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:46 smithi072 ceph-mon[30177]: pgmap v1078: 300 pgs: 300 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:47.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:46 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:16:47.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:46 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T02:16:47.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:46 smithi072 ceph-mon[30177]: osdmap e700: 8 total, 8 up, 8 in 2024-03-26T02:16:47.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:46 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T02:16:47.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:46 smithi072 ceph-mon[41619]: pgmap v1078: 300 pgs: 300 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:47.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:46 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:16:47.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:46 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T02:16:47.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:46 smithi072 ceph-mon[41619]: osdmap e700: 8 total, 8 up, 8 in 2024-03-26T02:16:47.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:46 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T02:16:47.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:46 smithi173 ceph-mon[33839]: pgmap v1078: 300 pgs: 300 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:47.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:46 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:16:47.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:46 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T02:16:47.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:46 smithi173 ceph-mon[33839]: osdmap e700: 8 total, 8 up, 8 in 2024-03-26T02:16:47.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:46 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T02:16:49.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:48 smithi072 ceph-mon[30177]: pgmap v1081: 300 pgs: 300 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:49.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:48 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T02:16:49.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:48 smithi072 ceph-mon[30177]: osdmap e701: 8 total, 8 up, 8 in 2024-03-26T02:16:49.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:48 smithi072 ceph-mon[41619]: pgmap v1081: 300 pgs: 300 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:49.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:48 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T02:16:49.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:48 smithi072 ceph-mon[41619]: osdmap e701: 8 total, 8 up, 8 in 2024-03-26T02:16:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:48 smithi173 ceph-mon[33839]: pgmap v1081: 300 pgs: 300 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:16:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:48 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T02:16:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:48 smithi173 ceph-mon[33839]: osdmap e701: 8 total, 8 up, 8 in 2024-03-26T02:16:51.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:50 smithi072 ceph-mon[30177]: pgmap v1083: 300 pgs: 300 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:16:51.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:50 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "327.2", "id": [3, 7]}]: dispatch 2024-03-26T02:16:51.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:50 smithi072 ceph-mon[41619]: pgmap v1083: 300 pgs: 300 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:16:51.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:50 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "327.2", "id": [3, 7]}]: dispatch 2024-03-26T02:16:51.260 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:50 smithi173 ceph-mon[33839]: pgmap v1083: 300 pgs: 300 active+clean; 4.4 MiB data, 710 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T02:16:51.260 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:50 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "327.2", "id": [3, 7]}]: dispatch 2024-03-26T02:16:51.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:16:51 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:16:52.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:51 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "327.2", "id": [3, 7]}]': finished 2024-03-26T02:16:52.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:51 smithi072 ceph-mon[30177]: osdmap e702: 8 total, 8 up, 8 in 2024-03-26T02:16:52.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:51 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:16:52.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:51 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "327.2", "id": [3, 7]}]': finished 2024-03-26T02:16:52.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:51 smithi072 ceph-mon[41619]: osdmap e702: 8 total, 8 up, 8 in 2024-03-26T02:16:52.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:51 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:16:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:51 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "327.2", "id": [3, 7]}]': finished 2024-03-26T02:16:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:51 smithi173 ceph-mon[33839]: osdmap e702: 8 total, 8 up, 8 in 2024-03-26T02:16:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:51 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:16:53.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:52 smithi072 ceph-mon[30177]: pgmap v1085: 300 pgs: 300 active+clean; 4.4 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 872 B/s rd, 4.1 KiB/s wr, 1 op/s 2024-03-26T02:16:53.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:52 smithi072 ceph-mon[30177]: osdmap e703: 8 total, 8 up, 8 in 2024-03-26T02:16:53.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:16:53.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:16:53.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:16:53.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:16:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:52 smithi072 ceph-mon[41619]: pgmap v1085: 300 pgs: 300 active+clean; 4.4 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 872 B/s rd, 4.1 KiB/s wr, 1 op/s 2024-03-26T02:16:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:52 smithi072 ceph-mon[41619]: osdmap e703: 8 total, 8 up, 8 in 2024-03-26T02:16:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:16:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:16:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:16:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:16:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:52 smithi173 ceph-mon[33839]: pgmap v1085: 300 pgs: 300 active+clean; 4.4 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 872 B/s rd, 4.1 KiB/s wr, 1 op/s 2024-03-26T02:16:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:52 smithi173 ceph-mon[33839]: osdmap e703: 8 total, 8 up, 8 in 2024-03-26T02:16:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:16:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:16:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:16:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:16:54.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:53 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:16:54.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:53 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:16:54.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:53 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:16:54.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:53 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:16:54.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:53 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:16:54.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:53 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:16:54.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:53 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:16:54.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:53 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:16:54.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:53 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:16:55.260 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:16:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:16:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:16:55.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:55 smithi173 ceph-mon[33839]: pgmap v1087: 300 pgs: 300 active+clean; 4.4 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-26T02:16:55.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:55 smithi072 ceph-mon[30177]: pgmap v1087: 300 pgs: 300 active+clean; 4.4 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-26T02:16:55.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:55 smithi072 ceph-mon[41619]: pgmap v1087: 300 pgs: 300 active+clean; 4.4 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-26T02:16:57.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:57 smithi173 ceph-mon[33839]: pgmap v1088: 300 pgs: 300 active+clean; 4.4 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 8.4 KiB/s wr, 2 op/s 2024-03-26T02:16:57.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:57 smithi072 ceph-mon[30177]: pgmap v1088: 300 pgs: 300 active+clean; 4.4 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 8.4 KiB/s wr, 2 op/s 2024-03-26T02:16:57.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:57 smithi072 ceph-mon[41619]: pgmap v1088: 300 pgs: 300 active+clean; 4.4 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 8.4 KiB/s wr, 2 op/s 2024-03-26T02:16:59.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:16:59 smithi173 ceph-mon[33839]: pgmap v1089: 300 pgs: 300 active+clean; 4.4 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 8.2 KiB/s wr, 2 op/s 2024-03-26T02:16:59.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:16:59 smithi072 ceph-mon[30177]: pgmap v1089: 300 pgs: 300 active+clean; 4.4 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 8.2 KiB/s wr, 2 op/s 2024-03-26T02:16:59.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:16:59 smithi072 ceph-mon[41619]: pgmap v1089: 300 pgs: 300 active+clean; 4.4 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 8.2 KiB/s wr, 2 op/s 2024-03-26T02:17:00.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:17:00.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-140"}]: dispatch 2024-03-26T02:17:00.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:17:00.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-140"}]: dispatch 2024-03-26T02:17:00.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:17:00.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-140"}]: dispatch 2024-03-26T02:17:01.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:01 smithi173 ceph-mon[33839]: pgmap v1090: 300 pgs: 300 active+clean; 4.4 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 577 B/s rd, 4.7 KiB/s wr, 1 op/s 2024-03-26T02:17:01.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:01 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-140"}]': finished 2024-03-26T02:17:01.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:01 smithi173 ceph-mon[33839]: osdmap e704: 8 total, 8 up, 8 in 2024-03-26T02:17:01.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:17:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:17:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:01 smithi072 ceph-mon[30177]: pgmap v1090: 300 pgs: 300 active+clean; 4.4 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 577 B/s rd, 4.7 KiB/s wr, 1 op/s 2024-03-26T02:17:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:01 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-140"}]': finished 2024-03-26T02:17:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:01 smithi072 ceph-mon[30177]: osdmap e704: 8 total, 8 up, 8 in 2024-03-26T02:17:01.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:01 smithi072 ceph-mon[41619]: pgmap v1090: 300 pgs: 300 active+clean; 4.4 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 577 B/s rd, 4.7 KiB/s wr, 1 op/s 2024-03-26T02:17:01.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:01 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-140"}]': finished 2024-03-26T02:17:01.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:01 smithi072 ceph-mon[41619]: osdmap e704: 8 total, 8 up, 8 in 2024-03-26T02:17:02.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:02 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:17:02.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:02 smithi173 ceph-mon[33839]: osdmap e705: 8 total, 8 up, 8 in 2024-03-26T02:17:02.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:02 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:17:02.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:02 smithi072 ceph-mon[30177]: osdmap e705: 8 total, 8 up, 8 in 2024-03-26T02:17:02.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:02 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:17:02.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:02 smithi072 ceph-mon[41619]: osdmap e705: 8 total, 8 up, 8 in 2024-03-26T02:17:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:03 smithi173 ceph-mon[33839]: pgmap v1093: 268 pgs: 268 active+clean; 4.4 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:17:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:03 smithi173 ceph-mon[33839]: osdmap e706: 8 total, 8 up, 8 in 2024-03-26T02:17:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:17:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:03 smithi072 ceph-mon[30177]: pgmap v1093: 268 pgs: 268 active+clean; 4.4 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:17:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:03 smithi072 ceph-mon[30177]: osdmap e706: 8 total, 8 up, 8 in 2024-03-26T02:17:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:17:03.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:03 smithi072 ceph-mon[41619]: pgmap v1093: 268 pgs: 268 active+clean; 4.4 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:17:03.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:03 smithi072 ceph-mon[41619]: osdmap e706: 8 total, 8 up, 8 in 2024-03-26T02:17:03.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:17:04.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:04 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:17:04.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:04 smithi173 ceph-mon[33839]: osdmap e707: 8 total, 8 up, 8 in 2024-03-26T02:17:04.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:04 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:17:04.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:04 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:17:04.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:04 smithi072 ceph-mon[30177]: osdmap e707: 8 total, 8 up, 8 in 2024-03-26T02:17:04.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:04 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:17:04.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:04 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:17:04.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:04 smithi072 ceph-mon[41619]: osdmap e707: 8 total, 8 up, 8 in 2024-03-26T02:17:04.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:04 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:17:05.307 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:17:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:17:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:17:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:05 smithi173 ceph-mon[33839]: pgmap v1096: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:17:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:17:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:05 smithi173 ceph-mon[33839]: osdmap e708: 8 total, 8 up, 8 in 2024-03-26T02:17:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-142"}]: dispatch 2024-03-26T02:17:05.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:05 smithi072 ceph-mon[41619]: pgmap v1096: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:17:05.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:17:05.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:05 smithi072 ceph-mon[41619]: osdmap e708: 8 total, 8 up, 8 in 2024-03-26T02:17:05.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-142"}]: dispatch 2024-03-26T02:17:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:05 smithi072 ceph-mon[30177]: pgmap v1096: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 711 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:17:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:17:05.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:05 smithi072 ceph-mon[30177]: osdmap e708: 8 total, 8 up, 8 in 2024-03-26T02:17:05.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-142"}]: dispatch 2024-03-26T02:17:06.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-142"}]': finished 2024-03-26T02:17:06.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:06 smithi173 ceph-mon[33839]: osdmap e709: 8 total, 8 up, 8 in 2024-03-26T02:17:06.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-142", "mode": "writeback"}]: dispatch 2024-03-26T02:17:06.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:06 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:17:06.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-142", "mode": "writeback"}]': finished 2024-03-26T02:17:06.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-142"}]': finished 2024-03-26T02:17:06.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:06 smithi072 ceph-mon[30177]: osdmap e709: 8 total, 8 up, 8 in 2024-03-26T02:17:06.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-142", "mode": "writeback"}]: dispatch 2024-03-26T02:17:06.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:06 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:17:06.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-142", "mode": "writeback"}]': finished 2024-03-26T02:17:06.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-142"}]': finished 2024-03-26T02:17:06.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:06 smithi072 ceph-mon[41619]: osdmap e709: 8 total, 8 up, 8 in 2024-03-26T02:17:06.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-142", "mode": "writeback"}]: dispatch 2024-03-26T02:17:06.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:06 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:17:06.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-142", "mode": "writeback"}]': finished 2024-03-26T02:17:07.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:07 smithi072 ceph-mon[30177]: pgmap v1099: 300 pgs: 300 active+clean; 4.4 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T02:17:07.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:07 smithi072 ceph-mon[30177]: osdmap e710: 8 total, 8 up, 8 in 2024-03-26T02:17:07.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:17:07.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_count","val": "2"}]': finished 2024-03-26T02:17:07.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:07 smithi072 ceph-mon[41619]: pgmap v1099: 300 pgs: 300 active+clean; 4.4 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T02:17:07.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:07 smithi072 ceph-mon[41619]: osdmap e710: 8 total, 8 up, 8 in 2024-03-26T02:17:07.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:17:07.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_count","val": "2"}]': finished 2024-03-26T02:17:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:07 smithi173 ceph-mon[33839]: pgmap v1099: 300 pgs: 300 active+clean; 4.4 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T02:17:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:07 smithi173 ceph-mon[33839]: osdmap e710: 8 total, 8 up, 8 in 2024-03-26T02:17:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:17:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_count","val": "2"}]': finished 2024-03-26T02:17:08.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:08 smithi072 ceph-mon[30177]: osdmap e711: 8 total, 8 up, 8 in 2024-03-26T02:17:08.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:17:08.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:17:08.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:17:08.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:17:08.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:08 smithi072 ceph-mon[30177]: osdmap e712: 8 total, 8 up, 8 in 2024-03-26T02:17:08.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:08 smithi072 ceph-mon[41619]: osdmap e711: 8 total, 8 up, 8 in 2024-03-26T02:17:08.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:17:08.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:17:08.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:17:08.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:17:08.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:08 smithi072 ceph-mon[41619]: osdmap e712: 8 total, 8 up, 8 in 2024-03-26T02:17:08.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:08 smithi173 ceph-mon[33839]: osdmap e711: 8 total, 8 up, 8 in 2024-03-26T02:17:08.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:17:08.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:17:08.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:17:08.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:17:08.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:08 smithi173 ceph-mon[33839]: osdmap e712: 8 total, 8 up, 8 in 2024-03-26T02:17:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:09 smithi072 ceph-mon[30177]: pgmap v1102: 300 pgs: 300 active+clean; 4.4 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T02:17:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:17:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:09 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:17:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T02:17:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:09 smithi072 ceph-mon[30177]: osdmap e713: 8 total, 8 up, 8 in 2024-03-26T02:17:09.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T02:17:09.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:09 smithi072 ceph-mon[41619]: pgmap v1102: 300 pgs: 300 active+clean; 4.4 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T02:17:09.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:17:09.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:09 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:17:09.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T02:17:09.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:09 smithi072 ceph-mon[41619]: osdmap e713: 8 total, 8 up, 8 in 2024-03-26T02:17:09.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T02:17:09.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:09 smithi173 ceph-mon[33839]: pgmap v1102: 300 pgs: 300 active+clean; 4.4 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T02:17:09.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:17:09.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:09 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:17:09.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T02:17:09.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:09 smithi173 ceph-mon[33839]: osdmap e713: 8 total, 8 up, 8 in 2024-03-26T02:17:09.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T02:17:11.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:11 smithi173 ceph-mon[33839]: pgmap v1105: 300 pgs: 300 active+clean; 4.4 MiB data, 716 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:17:11.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:11 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T02:17:11.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:11 smithi173 ceph-mon[33839]: osdmap e714: 8 total, 8 up, 8 in 2024-03-26T02:17:11.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:11 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T02:17:11.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:17:11 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:17:11.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:11 smithi072 ceph-mon[30177]: pgmap v1105: 300 pgs: 300 active+clean; 4.4 MiB data, 716 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:17:11.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:11 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T02:17:11.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:11 smithi072 ceph-mon[30177]: osdmap e714: 8 total, 8 up, 8 in 2024-03-26T02:17:11.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:11 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T02:17:11.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:11 smithi072 ceph-mon[41619]: pgmap v1105: 300 pgs: 300 active+clean; 4.4 MiB data, 716 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:17:11.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:11 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T02:17:11.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:11 smithi072 ceph-mon[41619]: osdmap e714: 8 total, 8 up, 8 in 2024-03-26T02:17:11.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:11 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T02:17:12.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:12 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:17:12.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T02:17:12.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:12 smithi072 ceph-mon[30177]: osdmap e715: 8 total, 8 up, 8 in 2024-03-26T02:17:12.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T02:17:12.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T02:17:12.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:12 smithi072 ceph-mon[30177]: osdmap e716: 8 total, 8 up, 8 in 2024-03-26T02:17:12.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:12 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:17:12.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T02:17:12.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:12 smithi072 ceph-mon[41619]: osdmap e715: 8 total, 8 up, 8 in 2024-03-26T02:17:12.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T02:17:12.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T02:17:12.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:12 smithi072 ceph-mon[41619]: osdmap e716: 8 total, 8 up, 8 in 2024-03-26T02:17:12.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:12 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:17:12.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T02:17:12.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:12 smithi173 ceph-mon[33839]: osdmap e715: 8 total, 8 up, 8 in 2024-03-26T02:17:12.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T02:17:12.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T02:17:12.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:12 smithi173 ceph-mon[33839]: osdmap e716: 8 total, 8 up, 8 in 2024-03-26T02:17:13.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:13 smithi072 ceph-mon[30177]: pgmap v1108: 300 pgs: 300 active+clean; 4.4 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:17:13.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:13 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:17:13.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:13 smithi072 ceph-mon[41619]: pgmap v1108: 300 pgs: 300 active+clean; 4.4 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:17:13.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:13 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:17:13.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:13 smithi173 ceph-mon[33839]: pgmap v1108: 300 pgs: 300 active+clean; 4.4 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:17:13.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:13 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:17:14.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:14 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:17:14.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:14 smithi072 ceph-mon[30177]: osdmap e717: 8 total, 8 up, 8 in 2024-03-26T02:17:14.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:14 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-142"}]: dispatch 2024-03-26T02:17:14.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:14 smithi072 ceph-mon[30177]: pgmap v1111: 300 pgs: 300 active+clean; 4.4 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:17:14.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:14 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:17:14.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:14 smithi072 ceph-mon[41619]: osdmap e717: 8 total, 8 up, 8 in 2024-03-26T02:17:14.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:14 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-142"}]: dispatch 2024-03-26T02:17:14.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:14 smithi072 ceph-mon[41619]: pgmap v1111: 300 pgs: 300 active+clean; 4.4 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:17:14.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:14 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:17:14.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:14 smithi173 ceph-mon[33839]: osdmap e717: 8 total, 8 up, 8 in 2024-03-26T02:17:14.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:14 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-142"}]: dispatch 2024-03-26T02:17:14.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:14 smithi173 ceph-mon[33839]: pgmap v1111: 300 pgs: 300 active+clean; 4.4 MiB data, 717 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:17:15.355 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:17:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:17:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:17:15.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-142"}]': finished 2024-03-26T02:17:15.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:15 smithi072 ceph-mon[41619]: osdmap e718: 8 total, 8 up, 8 in 2024-03-26T02:17:15.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:17:15.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-142"}]: dispatch 2024-03-26T02:17:15.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-142"}]': finished 2024-03-26T02:17:15.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:15 smithi072 ceph-mon[30177]: osdmap e718: 8 total, 8 up, 8 in 2024-03-26T02:17:15.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:17:15.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-142"}]: dispatch 2024-03-26T02:17:15.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-142"}]': finished 2024-03-26T02:17:15.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:15 smithi173 ceph-mon[33839]: osdmap e718: 8 total, 8 up, 8 in 2024-03-26T02:17:15.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:17:15.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-142"}]: dispatch 2024-03-26T02:17:16.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:16 smithi072 ceph-mon[30177]: osdmap e719: 8 total, 8 up, 8 in 2024-03-26T02:17:16.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:16 smithi072 ceph-mon[30177]: pgmap v1114: 268 pgs: 268 active+clean; 4.4 MiB data, 735 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T02:17:16.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:16 smithi072 ceph-mon[41619]: osdmap e719: 8 total, 8 up, 8 in 2024-03-26T02:17:16.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:16 smithi072 ceph-mon[41619]: pgmap v1114: 268 pgs: 268 active+clean; 4.4 MiB data, 735 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T02:17:16.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:16 smithi173 ceph-mon[33839]: osdmap e719: 8 total, 8 up, 8 in 2024-03-26T02:17:16.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:16 smithi173 ceph-mon[33839]: pgmap v1114: 268 pgs: 268 active+clean; 4.4 MiB data, 735 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T02:17:17.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:17 smithi072 ceph-mon[30177]: osdmap e720: 8 total, 8 up, 8 in 2024-03-26T02:17:17.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:17 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:17:17.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:17 smithi072 ceph-mon[41619]: osdmap e720: 8 total, 8 up, 8 in 2024-03-26T02:17:17.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:17 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:17:17.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:17 smithi173 ceph-mon[33839]: osdmap e720: 8 total, 8 up, 8 in 2024-03-26T02:17:17.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:17 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:17:18.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:17:18.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:18 smithi072 ceph-mon[30177]: osdmap e721: 8 total, 8 up, 8 in 2024-03-26T02:17:18.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:18 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:17:18.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:18 smithi072 ceph-mon[30177]: pgmap v1117: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 735 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T02:17:18.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:17:18.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:18 smithi072 ceph-mon[41619]: osdmap e721: 8 total, 8 up, 8 in 2024-03-26T02:17:18.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:18 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:17:18.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:18 smithi072 ceph-mon[41619]: pgmap v1117: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 735 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T02:17:18.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:17:18.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:18 smithi173 ceph-mon[33839]: osdmap e721: 8 total, 8 up, 8 in 2024-03-26T02:17:18.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:18 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:17:18.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:18 smithi173 ceph-mon[33839]: pgmap v1117: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 735 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T02:17:19.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:17:19.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:19 smithi072 ceph-mon[30177]: osdmap e722: 8 total, 8 up, 8 in 2024-03-26T02:17:19.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:19 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-144"}]: dispatch 2024-03-26T02:17:19.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:17:19.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:19 smithi072 ceph-mon[41619]: osdmap e722: 8 total, 8 up, 8 in 2024-03-26T02:17:19.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:19 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-144"}]: dispatch 2024-03-26T02:17:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:17:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:19 smithi173 ceph-mon[33839]: osdmap e722: 8 total, 8 up, 8 in 2024-03-26T02:17:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:19 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-144"}]: dispatch 2024-03-26T02:17:20.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-144"}]': finished 2024-03-26T02:17:20.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:20 smithi072 ceph-mon[30177]: osdmap e723: 8 total, 8 up, 8 in 2024-03-26T02:17:20.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:20 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-144", "mode": "readproxy"}]: dispatch 2024-03-26T02:17:20.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:20 smithi072 ceph-mon[30177]: pgmap v1120: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 735 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:17:20.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-144"}]': finished 2024-03-26T02:17:20.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:20 smithi072 ceph-mon[41619]: osdmap e723: 8 total, 8 up, 8 in 2024-03-26T02:17:20.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:20 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-144", "mode": "readproxy"}]: dispatch 2024-03-26T02:17:20.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:20 smithi072 ceph-mon[41619]: pgmap v1120: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 735 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:17:20.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-144"}]': finished 2024-03-26T02:17:20.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:20 smithi173 ceph-mon[33839]: osdmap e723: 8 total, 8 up, 8 in 2024-03-26T02:17:20.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:20 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-144", "mode": "readproxy"}]: dispatch 2024-03-26T02:17:20.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:20 smithi173 ceph-mon[33839]: pgmap v1120: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 735 MiB used, 715 GiB / 715 GiB avail 2024-03-26T02:17:21.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:17:21 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:17:21.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:21 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:17:21.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:21 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-144", "mode": "readproxy"}]': finished 2024-03-26T02:17:21.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:21 smithi173 ceph-mon[33839]: osdmap e724: 8 total, 8 up, 8 in 2024-03-26T02:17:21.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:21 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:17:21.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:21 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:17:21.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:21 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-144", "mode": "readproxy"}]': finished 2024-03-26T02:17:21.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:21 smithi072 ceph-mon[30177]: osdmap e724: 8 total, 8 up, 8 in 2024-03-26T02:17:21.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:21 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:17:21.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:21 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:17:21.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:21 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-144", "mode": "readproxy"}]': finished 2024-03-26T02:17:21.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:21 smithi072 ceph-mon[41619]: osdmap e724: 8 total, 8 up, 8 in 2024-03-26T02:17:21.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:21 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:17:22.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:22 smithi072 ceph-mon[30177]: pgmap v1122: 300 pgs: 300 active+clean; 4.4 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 233 B/s wr, 1 op/s 2024-03-26T02:17:22.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:22 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:17:22.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:22 smithi072 ceph-mon[41619]: pgmap v1122: 300 pgs: 300 active+clean; 4.4 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 233 B/s wr, 1 op/s 2024-03-26T02:17:22.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:22 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:17:22.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:22 smithi173 ceph-mon[33839]: pgmap v1122: 300 pgs: 300 active+clean; 4.4 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 233 B/s wr, 1 op/s 2024-03-26T02:17:22.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:22 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:17:25.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:24 smithi173 ceph-mon[33839]: pgmap v1123: 300 pgs: 300 active+clean; 4.4 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-26T02:17:25.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:24 smithi072 ceph-mon[30177]: pgmap v1123: 300 pgs: 300 active+clean; 4.4 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-26T02:17:25.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:24 smithi072 ceph-mon[41619]: pgmap v1123: 300 pgs: 300 active+clean; 4.4 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-26T02:17:25.220 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:17:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:17:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:17:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:26 smithi072 ceph-mon[30177]: pgmap v1124: 300 pgs: 300 active+clean; 4.4 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 138 B/s wr, 1 op/s 2024-03-26T02:17:27.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:26 smithi072 ceph-mon[41619]: pgmap v1124: 300 pgs: 300 active+clean; 4.4 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 138 B/s wr, 1 op/s 2024-03-26T02:17:27.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:26 smithi173 ceph-mon[33839]: pgmap v1124: 300 pgs: 300 active+clean; 4.4 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 138 B/s wr, 1 op/s 2024-03-26T02:17:29.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:28 smithi072 ceph-mon[30177]: pgmap v1125: 300 pgs: 300 active+clean; 4.4 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 122 B/s wr, 1 op/s 2024-03-26T02:17:29.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:28 smithi072 ceph-mon[41619]: pgmap v1125: 300 pgs: 300 active+clean; 4.4 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 122 B/s wr, 1 op/s 2024-03-26T02:17:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:28 smithi173 ceph-mon[33839]: pgmap v1125: 300 pgs: 300 active+clean; 4.4 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 122 B/s wr, 1 op/s 2024-03-26T02:17:31.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:30 smithi072 ceph-mon[30177]: pgmap v1126: 300 pgs: 300 active+clean; 4.4 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2024-03-26T02:17:31.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:17:31.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:30 smithi072 ceph-mon[41619]: pgmap v1126: 300 pgs: 300 active+clean; 4.4 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2024-03-26T02:17:31.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:17:31.291 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:30 smithi173 ceph-mon[33839]: pgmap v1126: 300 pgs: 300 active+clean; 4.4 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 102 B/s wr, 1 op/s 2024-03-26T02:17:31.291 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:17:31.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:17:31 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:17:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:31 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:17:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:31 smithi072 ceph-mon[30177]: osdmap e725: 8 total, 8 up, 8 in 2024-03-26T02:17:32.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:31 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-144"}]: dispatch 2024-03-26T02:17:32.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:31 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:17:32.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:31 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:17:32.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:31 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-144"}]': finished 2024-03-26T02:17:32.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:31 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:17:32.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:31 smithi072 ceph-mon[41619]: osdmap e725: 8 total, 8 up, 8 in 2024-03-26T02:17:32.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:31 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-144"}]: dispatch 2024-03-26T02:17:32.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:31 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:17:32.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:31 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:17:32.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:31 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-144"}]': finished 2024-03-26T02:17:32.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:31 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:17:32.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:31 smithi173 ceph-mon[33839]: osdmap e725: 8 total, 8 up, 8 in 2024-03-26T02:17:32.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:31 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-144"}]: dispatch 2024-03-26T02:17:32.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:31 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:17:32.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:31 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:17:32.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:31 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-144"}]': finished 2024-03-26T02:17:33.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:32 smithi072 ceph-mon[30177]: pgmap v1128: 300 pgs: 300 active+clean; 4.4 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:17:33.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:32 smithi072 ceph-mon[30177]: osdmap e726: 8 total, 8 up, 8 in 2024-03-26T02:17:33.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:17:33.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-144"}]: dispatch 2024-03-26T02:17:33.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:32 smithi072 ceph-mon[41619]: pgmap v1128: 300 pgs: 300 active+clean; 4.4 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:17:33.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:32 smithi072 ceph-mon[41619]: osdmap e726: 8 total, 8 up, 8 in 2024-03-26T02:17:33.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:17:33.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-144"}]: dispatch 2024-03-26T02:17:33.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:32 smithi173 ceph-mon[33839]: pgmap v1128: 300 pgs: 300 active+clean; 4.4 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:17:33.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:32 smithi173 ceph-mon[33839]: osdmap e726: 8 total, 8 up, 8 in 2024-03-26T02:17:33.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:17:33.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-144"}]: dispatch 2024-03-26T02:17:34.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:33 smithi072 ceph-mon[30177]: osdmap e727: 8 total, 8 up, 8 in 2024-03-26T02:17:34.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:33 smithi072 ceph-mon[41619]: osdmap e727: 8 total, 8 up, 8 in 2024-03-26T02:17:34.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:33 smithi173 ceph-mon[33839]: osdmap e727: 8 total, 8 up, 8 in 2024-03-26T02:17:35.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:17:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:17:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:17:35.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:34 smithi072 ceph-mon[30177]: pgmap v1131: 268 pgs: 268 active+clean; 4.4 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:17:35.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:34 smithi072 ceph-mon[30177]: osdmap e728: 8 total, 8 up, 8 in 2024-03-26T02:17:35.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:34 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:17:35.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:34 smithi072 ceph-mon[41619]: pgmap v1131: 268 pgs: 268 active+clean; 4.4 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:17:35.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:34 smithi072 ceph-mon[41619]: osdmap e728: 8 total, 8 up, 8 in 2024-03-26T02:17:35.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:34 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:17:35.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:34 smithi173 ceph-mon[33839]: pgmap v1131: 268 pgs: 268 active+clean; 4.4 MiB data, 753 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:17:35.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:34 smithi173 ceph-mon[33839]: osdmap e728: 8 total, 8 up, 8 in 2024-03-26T02:17:35.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:34 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:17:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:35 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:17:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:35 smithi072 ceph-mon[30177]: osdmap e729: 8 total, 8 up, 8 in 2024-03-26T02:17:36.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:35 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:17:36.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:35 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:17:36.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:35 smithi072 ceph-mon[41619]: osdmap e729: 8 total, 8 up, 8 in 2024-03-26T02:17:36.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:35 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:17:36.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:35 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:17:36.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:35 smithi173 ceph-mon[33839]: osdmap e729: 8 total, 8 up, 8 in 2024-03-26T02:17:36.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:35 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:17:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:36 smithi072 ceph-mon[30177]: pgmap v1134: 300 pgs: 300 active+clean; 4.4 MiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T02:17:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:36 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:17:37.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:36 smithi072 ceph-mon[30177]: osdmap e730: 8 total, 8 up, 8 in 2024-03-26T02:17:37.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:36 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-146"}]: dispatch 2024-03-26T02:17:37.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:36 smithi072 ceph-mon[41619]: pgmap v1134: 300 pgs: 300 active+clean; 4.4 MiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T02:17:37.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:36 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:17:37.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:36 smithi072 ceph-mon[41619]: osdmap e730: 8 total, 8 up, 8 in 2024-03-26T02:17:37.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:36 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-146"}]: dispatch 2024-03-26T02:17:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:36 smithi173 ceph-mon[33839]: pgmap v1134: 300 pgs: 300 active+clean; 4.4 MiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T02:17:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:36 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:17:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:36 smithi173 ceph-mon[33839]: osdmap e730: 8 total, 8 up, 8 in 2024-03-26T02:17:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:36 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-146"}]: dispatch 2024-03-26T02:17:38.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:37 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-146"}]': finished 2024-03-26T02:17:38.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:37 smithi072 ceph-mon[30177]: osdmap e731: 8 total, 8 up, 8 in 2024-03-26T02:17:38.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:37 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-146", "mode": "writeback"}]: dispatch 2024-03-26T02:17:38.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:37 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:17:38.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:37 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:17:38.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:37 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:17:38.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:37 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-146", "mode": "writeback"}]': finished 2024-03-26T02:17:38.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:37 smithi072 ceph-mon[30177]: osdmap e732: 8 total, 8 up, 8 in 2024-03-26T02:17:38.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:37 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-146"}]': finished 2024-03-26T02:17:38.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:37 smithi072 ceph-mon[41619]: osdmap e731: 8 total, 8 up, 8 in 2024-03-26T02:17:38.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:37 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-146", "mode": "writeback"}]: dispatch 2024-03-26T02:17:38.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:37 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:17:38.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:37 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:17:38.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:37 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:17:38.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:37 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-146", "mode": "writeback"}]': finished 2024-03-26T02:17:38.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:37 smithi072 ceph-mon[41619]: osdmap e732: 8 total, 8 up, 8 in 2024-03-26T02:17:38.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:37 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi072-82692-111", "overlaypool": "test-rados-api-smithi072-82692-146"}]': finished 2024-03-26T02:17:38.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:37 smithi173 ceph-mon[33839]: osdmap e731: 8 total, 8 up, 8 in 2024-03-26T02:17:38.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:37 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-146", "mode": "writeback"}]: dispatch 2024-03-26T02:17:38.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:37 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:17:38.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:37 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:17:38.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:37 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T02:17:38.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:37 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi072-82692-146", "mode": "writeback"}]': finished 2024-03-26T02:17:38.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:37 smithi173 ceph-mon[33839]: osdmap e732: 8 total, 8 up, 8 in 2024-03-26T02:17:39.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:38 smithi072 ceph-mon[30177]: pgmap v1137: 300 pgs: 300 active+clean; 4.4 MiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T02:17:39.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:38 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:17:39.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:38 smithi072 ceph-mon[41619]: pgmap v1137: 300 pgs: 300 active+clean; 4.4 MiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T02:17:39.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:38 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:17:39.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:38 smithi173 ceph-mon[33839]: pgmap v1137: 300 pgs: 300 active+clean; 4.4 MiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T02:17:39.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:38 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T02:17:40.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "hit_set_count","val": "2"}]': finished 2024-03-26T02:17:40.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:39 smithi072 ceph-mon[30177]: osdmap e733: 8 total, 8 up, 8 in 2024-03-26T02:17:40.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:39 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:17:40.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "hit_set_count","val": "2"}]': finished 2024-03-26T02:17:40.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:39 smithi072 ceph-mon[41619]: osdmap e733: 8 total, 8 up, 8 in 2024-03-26T02:17:40.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:39 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:17:40.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "hit_set_count","val": "2"}]': finished 2024-03-26T02:17:40.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:39 smithi173 ceph-mon[33839]: osdmap e733: 8 total, 8 up, 8 in 2024-03-26T02:17:40.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:39 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T02:17:41.214 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:40 smithi173 ceph-mon[33839]: pgmap v1140: 300 pgs: 300 active+clean; 4.4 MiB data, 754 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:17:41.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:17:41.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:40 smithi173 ceph-mon[33839]: osdmap e734: 8 total, 8 up, 8 in 2024-03-26T02:17:41.215 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:40 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:17:41.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:40 smithi072 ceph-mon[30177]: pgmap v1140: 300 pgs: 300 active+clean; 4.4 MiB data, 754 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:17:41.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:17:41.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:40 smithi072 ceph-mon[30177]: osdmap e734: 8 total, 8 up, 8 in 2024-03-26T02:17:41.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:40 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:17:41.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:40 smithi072 ceph-mon[41619]: pgmap v1140: 300 pgs: 300 active+clean; 4.4 MiB data, 754 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:17:41.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "hit_set_period","val": "600"}]': finished 2024-03-26T02:17:41.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:40 smithi072 ceph-mon[41619]: osdmap e734: 8 total, 8 up, 8 in 2024-03-26T02:17:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:40 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T02:17:41.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:17:41 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:17:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:41 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:17:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:41 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T02:17:42.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:41 smithi072 ceph-mon[30177]: osdmap e735: 8 total, 8 up, 8 in 2024-03-26T02:17:42.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:41 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T02:17:42.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:41 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:17:42.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:41 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T02:17:42.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:41 smithi072 ceph-mon[30177]: osdmap e736: 8 total, 8 up, 8 in 2024-03-26T02:17:42.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:41 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T02:17:42.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:41 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:17:42.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:41 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T02:17:42.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:41 smithi072 ceph-mon[41619]: osdmap e735: 8 total, 8 up, 8 in 2024-03-26T02:17:42.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:41 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T02:17:42.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:41 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:17:42.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:41 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T02:17:42.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:41 smithi072 ceph-mon[41619]: osdmap e736: 8 total, 8 up, 8 in 2024-03-26T02:17:42.221 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:41 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T02:17:42.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:41 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T02:17:42.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:41 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T02:17:42.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:41 smithi173 ceph-mon[33839]: osdmap e735: 8 total, 8 up, 8 in 2024-03-26T02:17:42.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:41 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T02:17:42.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:41 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:17:42.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:41 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T02:17:42.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:41 smithi173 ceph-mon[33839]: osdmap e736: 8 total, 8 up, 8 in 2024-03-26T02:17:42.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:41 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T02:17:43.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:42 smithi072 ceph-mon[30177]: pgmap v1143: 300 pgs: 300 active+clean; 4.4 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T02:17:43.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:42 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "target_max_objects","val": "1"}]': finished 2024-03-26T02:17:43.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:42 smithi072 ceph-mon[30177]: osdmap e737: 8 total, 8 up, 8 in 2024-03-26T02:17:43.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:42 smithi072 ceph-mon[41619]: pgmap v1143: 300 pgs: 300 active+clean; 4.4 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T02:17:43.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:42 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "target_max_objects","val": "1"}]': finished 2024-03-26T02:17:43.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:42 smithi072 ceph-mon[41619]: osdmap e737: 8 total, 8 up, 8 in 2024-03-26T02:17:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:42 smithi173 ceph-mon[33839]: pgmap v1143: 300 pgs: 300 active+clean; 4.4 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T02:17:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:42 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-146","var": "target_max_objects","val": "1"}]': finished 2024-03-26T02:17:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:42 smithi173 ceph-mon[33839]: osdmap e737: 8 total, 8 up, 8 in 2024-03-26T02:17:44.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:43 smithi072 ceph-mon[30177]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T02:17:44.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:43 smithi072 ceph-mon[41619]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T02:17:44.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:43 smithi173 ceph-mon[33839]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T02:17:45.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:44 smithi173 ceph-mon[33839]: pgmap v1146: 300 pgs: 300 active+clean; 4.4 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T02:17:45.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:44 smithi072 ceph-mon[30177]: pgmap v1146: 300 pgs: 300 active+clean; 4.4 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T02:17:45.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:44 smithi072 ceph-mon[41619]: pgmap v1146: 300 pgs: 300 active+clean; 4.4 MiB data, 738 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T02:17:45.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:17:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:17:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:17:47.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:46 smithi173 ceph-mon[33839]: pgmap v1147: 300 pgs: 300 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T02:17:47.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:46 smithi072 ceph-mon[30177]: pgmap v1147: 300 pgs: 300 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T02:17:47.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:46 smithi072 ceph-mon[41619]: pgmap v1147: 300 pgs: 300 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T02:17:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:48 smithi173 ceph-mon[33839]: pgmap v1148: 300 pgs: 300 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 745 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:17:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:48 smithi072 ceph-mon[30177]: pgmap v1148: 300 pgs: 300 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 745 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:17:49.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:48 smithi072 ceph-mon[41619]: pgmap v1148: 300 pgs: 300 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 745 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:17:51.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:50 smithi173 ceph-mon[33839]: pgmap v1149: 300 pgs: 300 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-03-26T02:17:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:50 smithi072 ceph-mon[30177]: pgmap v1149: 300 pgs: 300 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-03-26T02:17:51.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:50 smithi072 ceph-mon[41619]: pgmap v1149: 300 pgs: 300 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-03-26T02:17:51.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:17:51 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:17:52.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:51 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:17:52.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:51 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:17:52.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:51 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:17:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:52 smithi173 ceph-mon[33839]: pgmap v1150: 300 pgs: 300 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:17:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:17:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:17:53.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:52 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:17:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:52 smithi072 ceph-mon[30177]: pgmap v1150: 300 pgs: 300 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:17:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:17:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:17:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:52 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:17:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:52 smithi072 ceph-mon[41619]: pgmap v1150: 300 pgs: 300 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:17:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:17:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:17:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:52 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:17:54.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:53 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:17:54.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:53 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:17:54.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:53 smithi173 ceph-mon[33839]: osdmap e738: 8 total, 8 up, 8 in 2024-03-26T02:17:54.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:53 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-146"}]: dispatch 2024-03-26T02:17:54.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:53 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:17:54.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:53 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:17:54.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:53 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:17:54.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:53 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:17:54.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:53 smithi072 ceph-mon[41619]: osdmap e738: 8 total, 8 up, 8 in 2024-03-26T02:17:54.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:53 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-146"}]: dispatch 2024-03-26T02:17:54.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:53 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:17:54.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:53 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:17:54.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:53 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:17:54.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:53 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:17:54.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:53 smithi072 ceph-mon[30177]: osdmap e738: 8 total, 8 up, 8 in 2024-03-26T02:17:54.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:53 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-146"}]: dispatch 2024-03-26T02:17:54.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:53 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:17:54.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:53 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:17:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:54 smithi173 ceph-mon[33839]: pgmap v1152: 300 pgs: 300 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:17:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:54 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-146"}]': finished 2024-03-26T02:17:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:54 smithi173 ceph-mon[33839]: osdmap e739: 8 total, 8 up, 8 in 2024-03-26T02:17:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:54 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:17:55.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:54 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-146"}]: dispatch 2024-03-26T02:17:55.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:54 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:17:55.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:54 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:17:55.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:54 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:17:55.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:17:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:17:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:17:55.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:54 smithi072 ceph-mon[30177]: pgmap v1152: 300 pgs: 300 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:17:55.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:54 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-146"}]': finished 2024-03-26T02:17:55.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:54 smithi072 ceph-mon[30177]: osdmap e739: 8 total, 8 up, 8 in 2024-03-26T02:17:55.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:54 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:17:55.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:54 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-146"}]: dispatch 2024-03-26T02:17:55.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:54 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:17:55.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:54 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:17:55.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:54 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:17:55.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:54 smithi072 ceph-mon[41619]: pgmap v1152: 300 pgs: 300 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T02:17:55.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:54 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-146"}]': finished 2024-03-26T02:17:55.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:54 smithi072 ceph-mon[41619]: osdmap e739: 8 total, 8 up, 8 in 2024-03-26T02:17:55.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:54 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:17:55.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:54 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-146"}]: dispatch 2024-03-26T02:17:55.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:54 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:17:55.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:54 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:17:55.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:54 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:17:56.313 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:17:56 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=sqlstore.transactions t=2024-03-26T02:17:56.014062481Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T02:17:56.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:56 smithi072 ceph-mon[30177]: osdmap e740: 8 total, 8 up, 8 in 2024-03-26T02:17:56.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:56 smithi072 ceph-mon[30177]: pgmap v1155: 268 pgs: 268 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:17:56.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:56 smithi072 ceph-mon[41619]: osdmap e740: 8 total, 8 up, 8 in 2024-03-26T02:17:56.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:56 smithi072 ceph-mon[41619]: pgmap v1155: 268 pgs: 268 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:17:56.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:56 smithi173 ceph-mon[33839]: osdmap e740: 8 total, 8 up, 8 in 2024-03-26T02:17:56.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:56 smithi173 ceph-mon[33839]: pgmap v1155: 268 pgs: 268 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:17:57.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:57 smithi072 ceph-mon[30177]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T02:17:57.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:57 smithi072 ceph-mon[30177]: osdmap e741: 8 total, 8 up, 8 in 2024-03-26T02:17:57.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:57 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:17:57.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:57 smithi072 ceph-mon[41619]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T02:17:57.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:57 smithi072 ceph-mon[41619]: osdmap e741: 8 total, 8 up, 8 in 2024-03-26T02:17:57.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:57 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:17:57.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:57 smithi173 ceph-mon[33839]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T02:17:57.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:57 smithi173 ceph-mon[33839]: osdmap e741: 8 total, 8 up, 8 in 2024-03-26T02:17:57.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:57 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:17:58.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:17:58.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:58 smithi072 ceph-mon[30177]: osdmap e742: 8 total, 8 up, 8 in 2024-03-26T02:17:58.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:58 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:17:58.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:58 smithi072 ceph-mon[30177]: pgmap v1158: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:17:58.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:17:58.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:58 smithi072 ceph-mon[41619]: osdmap e742: 8 total, 8 up, 8 in 2024-03-26T02:17:58.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:58 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:17:58.721 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:58 smithi072 ceph-mon[41619]: pgmap v1158: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:17:58.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:17:58.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:58 smithi173 ceph-mon[33839]: osdmap e742: 8 total, 8 up, 8 in 2024-03-26T02:17:58.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:58 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T02:17:58.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:58 smithi173 ceph-mon[33839]: pgmap v1158: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T02:17:59.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:59 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:17:59.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:59 smithi072 ceph-mon[30177]: osdmap e743: 8 total, 8 up, 8 in 2024-03-26T02:17:59.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:17:59 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-148"}]: dispatch 2024-03-26T02:17:59.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:59 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:17:59.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:59 smithi072 ceph-mon[41619]: osdmap e743: 8 total, 8 up, 8 in 2024-03-26T02:17:59.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:17:59 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-148"}]: dispatch 2024-03-26T02:17:59.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:59 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T02:17:59.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:59 smithi173 ceph-mon[33839]: osdmap e743: 8 total, 8 up, 8 in 2024-03-26T02:17:59.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:17:59 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-148"}]: dispatch 2024-03-26T02:18:00.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-148"}]': finished 2024-03-26T02:18:00.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:00 smithi072 ceph-mon[30177]: osdmap e744: 8 total, 8 up, 8 in 2024-03-26T02:18:00.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:18:00.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:00 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-148"}]: dispatch 2024-03-26T02:18:00.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:00 smithi072 ceph-mon[30177]: pgmap v1161: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:18:00.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-148"}]': finished 2024-03-26T02:18:00.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:00 smithi072 ceph-mon[41619]: osdmap e744: 8 total, 8 up, 8 in 2024-03-26T02:18:00.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:18:00.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:00 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-148"}]: dispatch 2024-03-26T02:18:00.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:00 smithi072 ceph-mon[41619]: pgmap v1161: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:18:00.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-148"}]': finished 2024-03-26T02:18:00.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:00 smithi173 ceph-mon[33839]: osdmap e744: 8 total, 8 up, 8 in 2024-03-26T02:18:00.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:18:00.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:00 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-148"}]: dispatch 2024-03-26T02:18:00.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:00 smithi173 ceph-mon[33839]: pgmap v1161: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 739 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:18:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:01 smithi072 ceph-mon[30177]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:18:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:01 smithi072 ceph-mon[30177]: osdmap e745: 8 total, 8 up, 8 in 2024-03-26T02:18:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:01 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:18:01.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:01 smithi072 ceph-mon[41619]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:18:01.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:01 smithi072 ceph-mon[41619]: osdmap e745: 8 total, 8 up, 8 in 2024-03-26T02:18:01.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:01 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:18:01.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:18:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:18:01.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:01 smithi173 ceph-mon[33839]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:18:01.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:01 smithi173 ceph-mon[33839]: osdmap e745: 8 total, 8 up, 8 in 2024-03-26T02:18:01.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:01 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:18:02.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:02 smithi072 ceph-mon[30177]: osdmap e746: 8 total, 8 up, 8 in 2024-03-26T02:18:02.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:02 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:18:02.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:02 smithi072 ceph-mon[30177]: pgmap v1164: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 740 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T02:18:02.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:02 smithi072 ceph-mon[41619]: osdmap e746: 8 total, 8 up, 8 in 2024-03-26T02:18:02.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:02 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:18:02.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:02 smithi072 ceph-mon[41619]: pgmap v1164: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 740 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T02:18:02.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:02 smithi173 ceph-mon[33839]: osdmap e746: 8 total, 8 up, 8 in 2024-03-26T02:18:02.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:02 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:18:02.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:02 smithi173 ceph-mon[33839]: pgmap v1164: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 740 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T02:18:03.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:18:03.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:03 smithi072 ceph-mon[41619]: osdmap e747: 8 total, 8 up, 8 in 2024-03-26T02:18:03.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:18:03.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:03 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-150"}]: dispatch 2024-03-26T02:18:03.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:18:03.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:03 smithi072 ceph-mon[30177]: osdmap e747: 8 total, 8 up, 8 in 2024-03-26T02:18:03.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:18:03.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:03 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-150"}]: dispatch 2024-03-26T02:18:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:18:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:03 smithi173 ceph-mon[33839]: osdmap e747: 8 total, 8 up, 8 in 2024-03-26T02:18:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:18:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:03 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-150"}]: dispatch 2024-03-26T02:18:04.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:04 smithi173 ceph-mon[33839]: osdmap e748: 8 total, 8 up, 8 in 2024-03-26T02:18:04.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:04 smithi173 ceph-mon[33839]: pgmap v1167: 268 pgs: 268 active+clean; 4.4 MiB data, 740 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T02:18:04.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:04 smithi072 ceph-mon[30177]: osdmap e748: 8 total, 8 up, 8 in 2024-03-26T02:18:04.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:04 smithi072 ceph-mon[30177]: pgmap v1167: 268 pgs: 268 active+clean; 4.4 MiB data, 740 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T02:18:04.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:04 smithi072 ceph-mon[41619]: osdmap e748: 8 total, 8 up, 8 in 2024-03-26T02:18:04.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:04 smithi072 ceph-mon[41619]: pgmap v1167: 268 pgs: 268 active+clean; 4.4 MiB data, 740 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T02:18:05.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:18:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:18:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:18:05.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:05 smithi173 ceph-mon[33839]: osdmap e749: 8 total, 8 up, 8 in 2024-03-26T02:18:05.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:05 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:18:05.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:05 smithi072 ceph-mon[30177]: osdmap e749: 8 total, 8 up, 8 in 2024-03-26T02:18:05.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:05 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:18:05.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:05 smithi072 ceph-mon[41619]: osdmap e749: 8 total, 8 up, 8 in 2024-03-26T02:18:05.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:05 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:18:06.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:18:06.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:06 smithi173 ceph-mon[33839]: osdmap e750: 8 total, 8 up, 8 in 2024-03-26T02:18:06.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:18:06.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-152"}]: dispatch 2024-03-26T02:18:06.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:06 smithi173 ceph-mon[33839]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:18:06.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:06 smithi173 ceph-mon[33839]: pgmap v1170: 300 pgs: 5 creating+peering, 295 active+clean; 4.4 MiB data, 744 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-26T02:18:06.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:18:06.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:06 smithi072 ceph-mon[41619]: osdmap e750: 8 total, 8 up, 8 in 2024-03-26T02:18:06.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:18:06.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-152"}]: dispatch 2024-03-26T02:18:06.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:06 smithi072 ceph-mon[41619]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:18:06.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:06 smithi072 ceph-mon[41619]: pgmap v1170: 300 pgs: 5 creating+peering, 295 active+clean; 4.4 MiB data, 744 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-26T02:18:06.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:18:06.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:06 smithi072 ceph-mon[30177]: osdmap e750: 8 total, 8 up, 8 in 2024-03-26T02:18:06.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:18:06.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-152"}]: dispatch 2024-03-26T02:18:06.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:06 smithi072 ceph-mon[30177]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:18:06.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:06 smithi072 ceph-mon[30177]: pgmap v1170: 300 pgs: 5 creating+peering, 295 active+clean; 4.4 MiB data, 744 MiB used, 714 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-26T02:18:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:07 smithi173 ceph-mon[33839]: osdmap e751: 8 total, 8 up, 8 in 2024-03-26T02:18:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:07 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:18:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:07 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:18:07.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:07 smithi072 ceph-mon[41619]: osdmap e751: 8 total, 8 up, 8 in 2024-03-26T02:18:07.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:07 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:18:07.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:07 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:18:07.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:07 smithi072 ceph-mon[30177]: osdmap e751: 8 total, 8 up, 8 in 2024-03-26T02:18:07.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:07 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:18:07.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:07 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:18:08.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:08 smithi173 ceph-mon[33839]: osdmap e752: 8 total, 8 up, 8 in 2024-03-26T02:18:08.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:18:08.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:08 smithi173 ceph-mon[33839]: pgmap v1173: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 744 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-03-26T02:18:08.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:08 smithi072 ceph-mon[30177]: osdmap e752: 8 total, 8 up, 8 in 2024-03-26T02:18:08.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:18:08.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:08 smithi072 ceph-mon[30177]: pgmap v1173: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 744 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-03-26T02:18:08.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:08 smithi072 ceph-mon[41619]: osdmap e752: 8 total, 8 up, 8 in 2024-03-26T02:18:08.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T02:18:08.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:08 smithi072 ceph-mon[41619]: pgmap v1173: 300 pgs: 32 unknown, 268 active+clean; 4.4 MiB data, 744 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-03-26T02:18:09.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:18:09.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:09 smithi173 ceph-mon[33839]: osdmap e753: 8 total, 8 up, 8 in 2024-03-26T02:18:09.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-111","var": "dedup_tier","val": "test-rados-api-smithi072-82692-154"}]: dispatch 2024-03-26T02:18:09.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:18:09.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:09 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-154"}]: dispatch 2024-03-26T02:18:09.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:18:09.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:09 smithi072 ceph-mon[30177]: osdmap e753: 8 total, 8 up, 8 in 2024-03-26T02:18:09.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-111","var": "dedup_tier","val": "test-rados-api-smithi072-82692-154"}]: dispatch 2024-03-26T02:18:09.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:18:09.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:09 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-154"}]: dispatch 2024-03-26T02:18:09.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi072-82692-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T02:18:09.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:09 smithi072 ceph-mon[41619]: osdmap e753: 8 total, 8 up, 8 in 2024-03-26T02:18:09.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi072-82692-111","var": "dedup_tier","val": "test-rados-api-smithi072-82692-154"}]: dispatch 2024-03-26T02:18:09.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:18:09.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:09 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi072-82692-111", "tierpool": "test-rados-api-smithi072-82692-154"}]: dispatch 2024-03-26T02:18:10.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:10 smithi173 ceph-mon[33839]: osdmap e754: 8 total, 8 up, 8 in 2024-03-26T02:18:10.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:10 smithi173 ceph-mon[33839]: pgmap v1176: 268 pgs: 268 active+clean; 4.4 MiB data, 744 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:18:10.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:10 smithi072 ceph-mon[30177]: osdmap e754: 8 total, 8 up, 8 in 2024-03-26T02:18:10.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:10 smithi072 ceph-mon[30177]: pgmap v1176: 268 pgs: 268 active+clean; 4.4 MiB data, 744 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:18:10.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:10 smithi072 ceph-mon[41619]: osdmap e754: 8 total, 8 up, 8 in 2024-03-26T02:18:10.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:10 smithi072 ceph-mon[41619]: pgmap v1176: 268 pgs: 268 active+clean; 4.4 MiB data, 744 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:18:11.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:18:11 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:18:11.715 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:11 smithi173 ceph-mon[33839]: osdmap e755: 8 total, 8 up, 8 in 2024-03-26T02:18:11.715 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:11 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:18:11.715 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:11 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:18:11.870 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:11 smithi072 ceph-mon[30177]: osdmap e755: 8 total, 8 up, 8 in 2024-03-26T02:18:11.871 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:11 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:18:11.871 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:11 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:18:11.871 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:11 smithi072 ceph-mon[41619]: osdmap e755: 8 total, 8 up, 8 in 2024-03-26T02:18:11.871 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:11 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:18:11.871 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:11 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:18:12.538 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (7827 ms) 2024-03-26T02:18:12.538 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-03-26T02:18:12.538 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13063 ms) 2024-03-26T02:18:12.538 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-03-26T02:18:12.538 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (8042 ms) 2024-03-26T02:18:12.538 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-03-26T02:18:12.538 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (13084 ms) 2024-03-26T02:18:12.538 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-03-26T02:18:12.538 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (7819 ms) 2024-03-26T02:18:12.538 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-03-26T02:18:12.538 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8056 ms) 2024-03-26T02:18:12.538 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-03-26T02:18:12.539 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-26T02:18:12.539 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: ok, hit_set contains 326:602f83fe:::foo:head 2024-03-26T02:18:12.539 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9054 ms) 2024-03-26T02:18:12.539 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-03-26T02:18:12.539 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: got ls 1711419408,0 2024-03-26T02:18:12.539 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: first is 1711419408 2024-03-26T02:18:12.539 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: got ls 1711419408,0 2024-03-26T02:18:12.539 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: got ls 1711419408,0 2024-03-26T02:18:12.539 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: got ls 1711419408,1711419410,1711419411,0 2024-03-26T02:18:12.539 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: got ls 1711419408,1711419410,1711419411,0 2024-03-26T02:18:12.539 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: got ls 1711419408,1711419410,1711419411,0 2024-03-26T02:18:12.539 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: got ls 1711419408,1711419410,1711419411,1711419413,1711419414,0 2024-03-26T02:18:12.539 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: got ls 1711419408,1711419410,1711419411,1711419413,1711419414,0 2024-03-26T02:18:12.539 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: got ls 1711419408,1711419410,1711419411,1711419413,1711419414,0 2024-03-26T02:18:12.539 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: got ls 1711419408,1711419410,1711419411,1711419413,1711419414,1711419416,1711419417,0 2024-03-26T02:18:12.540 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: got ls 1711419408,1711419410,1711419411,1711419413,1711419414,1711419416,1711419417,0 2024-03-26T02:18:12.540 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: got ls 1711419408,1711419410,1711419411,1711419413,1711419414,1711419416,1711419417,0 2024-03-26T02:18:12.540 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: got ls 1711419411,1711419413,1711419414,1711419416,1711419417,1711419419,1711419420,0 2024-03-26T02:18:12.540 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: first now 1711419411, trimmed 2024-03-26T02:18:12.540 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (20414 ms) 2024-03-26T02:18:12.540 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-03-26T02:18:12.540 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: foo0 2024-03-26T02:18:12.540 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-26T02:18:12.540 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (14070 ms) 2024-03-26T02:18:12.540 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-03-26T02:18:12.540 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (17494 ms) 2024-03-26T02:18:12.540 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-03-26T02:18:12.540 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (22576 ms) 2024-03-26T02:18:12.540 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-03-26T02:18:12.540 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5032 ms) 2024-03-26T02:18:12.540 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-03-26T02:18:12.540 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3015 ms) 2024-03-26T02:18:12.541 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-03-26T02:18:12.541 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3032 ms) 2024-03-26T02:18:12.541 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-03-26T02:18:12.541 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3012 ms) 2024-03-26T02:18:12.541 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (231175 ms total) 2024-03-26T02:18:12.541 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: 2024-03-26T02:18:12.541 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-03-26T02:18:12.541 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (863348 ms total) 2024-03-26T02:18:12.541 INFO:tasks.workunit.client.0.smithi072.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-03-26T02:18:12.544 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:18:12.544 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82360 2024-03-26T02:18:12.544 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 82360 2024-03-26T02:18:12.544 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:18:12.544 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82575 2024-03-26T02:18:12.544 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 82575 2024-03-26T02:18:12.544 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:18:12.544 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83137 2024-03-26T02:18:12.545 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 83137 2024-03-26T02:18:12.545 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:18:12.545 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82870 2024-03-26T02:18:12.545 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 82870 2024-03-26T02:18:12.545 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:18:12.545 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82696 2024-03-26T02:18:12.545 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 82696 2024-03-26T02:18:12.545 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:18:12.545 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82441 2024-03-26T02:18:12.545 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 82441 2024-03-26T02:18:12.545 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:18:12.545 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82783 2024-03-26T02:18:12.545 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 82783 2024-03-26T02:18:12.545 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:18:12.545 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83262 2024-03-26T02:18:12.545 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 83262 2024-03-26T02:18:12.545 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:18:12.546 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83321 2024-03-26T02:18:12.546 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 83321 2024-03-26T02:18:12.546 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:18:12.546 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82459 2024-03-26T02:18:12.546 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 82459 2024-03-26T02:18:12.546 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:18:12.546 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83151 2024-03-26T02:18:12.546 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 83151 2024-03-26T02:18:12.546 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:18:12.546 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83043 2024-03-26T02:18:12.546 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 83043 2024-03-26T02:18:12.546 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:18:12.546 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82334 2024-03-26T02:18:12.546 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 82334 2024-03-26T02:18:12.546 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:18:12.546 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82397 2024-03-26T02:18:12.547 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 82397 2024-03-26T02:18:12.547 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:18:12.547 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83076 2024-03-26T02:18:12.547 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 83076 2024-03-26T02:18:12.547 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:18:12.547 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82325 2024-03-26T02:18:12.547 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 82325 2024-03-26T02:18:12.547 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:18:12.547 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82345 2024-03-26T02:18:12.547 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 82345 2024-03-26T02:18:12.547 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:18:12.547 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83484 2024-03-26T02:18:12.547 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 83484 2024-03-26T02:18:12.547 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:18:12.547 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82611 2024-03-26T02:18:12.547 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 82611 2024-03-26T02:18:12.548 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:18:12.548 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83528 2024-03-26T02:18:12.548 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 83528 2024-03-26T02:18:12.548 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:18:12.548 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82726 2024-03-26T02:18:12.548 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 82726 2024-03-26T02:18:12.548 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:18:12.548 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83208 2024-03-26T02:18:12.548 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 83208 2024-03-26T02:18:12.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:18:12.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:12 smithi173 ceph-mon[33839]: osdmap e756: 8 total, 8 up, 8 in 2024-03-26T02:18:12.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:18:12.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:12 smithi173 ceph-mon[33839]: pgmap v1179: 260 pgs: 260 active+clean; 582 KiB data, 749 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:18:12.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:18:12.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:18:12.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:12 smithi072 ceph-mon[30177]: osdmap e756: 8 total, 8 up, 8 in 2024-03-26T02:18:12.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:18:12.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:12 smithi072 ceph-mon[30177]: pgmap v1179: 260 pgs: 260 active+clean; 582 KiB data, 749 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:18:12.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:18:12.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:18:12.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:12 smithi072 ceph-mon[41619]: osdmap e756: 8 total, 8 up, 8 in 2024-03-26T02:18:12.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82692-111"}]: dispatch 2024-03-26T02:18:12.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:12 smithi072 ceph-mon[41619]: pgmap v1179: 260 pgs: 260 active+clean; 582 KiB data, 749 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:18:12.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/447121809' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi072-82692-111"}]': finished 2024-03-26T02:18:13.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:13 smithi173 ceph-mon[33839]: osdmap e757: 8 total, 8 up, 8 in 2024-03-26T02:18:13.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:13 smithi173 ceph-mon[33839]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:18:13.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:13 smithi072 ceph-mon[30177]: osdmap e757: 8 total, 8 up, 8 in 2024-03-26T02:18:13.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:13 smithi072 ceph-mon[30177]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:18:13.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:13 smithi072 ceph-mon[41619]: osdmap e757: 8 total, 8 up, 8 in 2024-03-26T02:18:13.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:13 smithi072 ceph-mon[41619]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:18:14.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:14 smithi173 ceph-mon[33839]: pgmap v1181: 260 pgs: 260 active+clean; 582 KiB data, 749 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:18:14.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:14 smithi072 ceph-mon[30177]: pgmap v1181: 260 pgs: 260 active+clean; 582 KiB data, 749 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:18:14.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:14 smithi072 ceph-mon[41619]: pgmap v1181: 260 pgs: 260 active+clean; 582 KiB data, 749 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:18:15.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:18:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:18:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:18:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:16 smithi072 ceph-mon[30177]: pgmap v1182: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:18:17.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:16 smithi072 ceph-mon[41619]: pgmap v1182: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:18:17.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:16 smithi173 ceph-mon[33839]: pgmap v1182: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:18:19.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:18 smithi072 ceph-mon[30177]: pgmap v1183: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 705 B/s rd, 0 op/s 2024-03-26T02:18:19.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:18 smithi072 ceph-mon[41619]: pgmap v1183: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 705 B/s rd, 0 op/s 2024-03-26T02:18:19.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:18 smithi173 ceph-mon[33839]: pgmap v1183: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 705 B/s rd, 0 op/s 2024-03-26T02:18:21.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:20 smithi072 ceph-mon[30177]: pgmap v1184: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 620 B/s rd, 0 op/s 2024-03-26T02:18:21.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:20 smithi072 ceph-mon[41619]: pgmap v1184: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 620 B/s rd, 0 op/s 2024-03-26T02:18:21.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:20 smithi173 ceph-mon[33839]: pgmap v1184: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 620 B/s rd, 0 op/s 2024-03-26T02:18:21.814 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:18:21 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:18:22.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:21 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:18:22.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:21 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:18:22.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:21 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:18:23.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:22 smithi072 ceph-mon[30177]: pgmap v1185: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:18:23.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:22 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:18:23.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:22 smithi072 ceph-mon[41619]: pgmap v1185: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:18:23.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:22 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:18:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:22 smithi173 ceph-mon[33839]: pgmap v1185: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:18:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:22 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:18:25.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:24 smithi072 ceph-mon[30177]: pgmap v1186: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-03-26T02:18:25.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:24 smithi072 ceph-mon[41619]: pgmap v1186: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-03-26T02:18:25.220 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:18:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:18:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:18:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:24 smithi173 ceph-mon[33839]: pgmap v1186: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 910 B/s rd, 0 op/s 2024-03-26T02:18:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:26 smithi072 ceph-mon[30177]: pgmap v1187: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:18:27.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:26 smithi072 ceph-mon[41619]: pgmap v1187: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:18:27.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:26 smithi173 ceph-mon[33839]: pgmap v1187: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:18:29.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:28 smithi072 ceph-mon[30177]: pgmap v1188: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:18:29.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:28 smithi072 ceph-mon[41619]: pgmap v1188: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:18:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:28 smithi173 ceph-mon[33839]: pgmap v1188: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:18:31.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:30 smithi072 ceph-mon[30177]: pgmap v1189: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:18:31.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:30 smithi072 ceph-mon[41619]: pgmap v1189: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:18:31.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:30 smithi173 ceph-mon[33839]: pgmap v1189: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:18:31.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:18:31 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:18:32.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:31 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:18:32.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:31 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:18:32.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:31 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:18:33.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:32 smithi072 ceph-mon[30177]: pgmap v1190: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:18:33.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:32 smithi072 ceph-mon[41619]: pgmap v1190: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:18:33.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:32 smithi173 ceph-mon[33839]: pgmap v1190: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:18:35.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:34 smithi072 ceph-mon[30177]: pgmap v1191: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:18:35.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:18:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:18:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:18:35.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:34 smithi072 ceph-mon[41619]: pgmap v1191: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:18:35.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:34 smithi173 ceph-mon[33839]: pgmap v1191: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:18:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:36 smithi072 ceph-mon[30177]: pgmap v1192: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:18:37.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:36 smithi072 ceph-mon[41619]: pgmap v1192: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:18:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:36 smithi173 ceph-mon[33839]: pgmap v1192: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:18:38.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:37 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:18:38.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:37 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:18:38.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:37 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:18:39.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:38 smithi072 ceph-mon[30177]: pgmap v1193: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:18:39.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:38 smithi072 ceph-mon[41619]: pgmap v1193: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:18:39.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:38 smithi173 ceph-mon[33839]: pgmap v1193: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:18:41.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:40 smithi072 ceph-mon[30177]: pgmap v1194: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:18:41.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:40 smithi072 ceph-mon[41619]: pgmap v1194: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:18:41.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:40 smithi173 ceph-mon[33839]: pgmap v1194: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:18:41.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:18:41 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:18:43.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:42 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:18:43.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:42 smithi072 ceph-mon[30177]: pgmap v1195: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:18:43.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:42 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:18:43.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:42 smithi072 ceph-mon[41619]: pgmap v1195: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:18:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:42 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:18:43.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:42 smithi173 ceph-mon[33839]: pgmap v1195: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:18:45.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:44 smithi072 ceph-mon[30177]: pgmap v1196: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:18:45.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:44 smithi072 ceph-mon[41619]: pgmap v1196: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:18:45.220 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:18:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:18:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:18:45.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:44 smithi173 ceph-mon[33839]: pgmap v1196: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:18:47.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:46 smithi072 ceph-mon[30177]: pgmap v1197: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:18:47.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:46 smithi072 ceph-mon[41619]: pgmap v1197: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:18:47.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:46 smithi173 ceph-mon[33839]: pgmap v1197: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:18:49.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:48 smithi072 ceph-mon[30177]: pgmap v1198: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:18:49.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:48 smithi072 ceph-mon[41619]: pgmap v1198: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:18:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:48 smithi173 ceph-mon[33839]: pgmap v1198: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:18:51.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:50 smithi072 ceph-mon[30177]: pgmap v1199: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:18:51.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:50 smithi072 ceph-mon[41619]: pgmap v1199: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:18:51.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:50 smithi173 ceph-mon[33839]: pgmap v1199: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:18:51.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:18:51 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:18:53.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:52 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:18:53.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:52 smithi072 ceph-mon[30177]: pgmap v1200: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:18:53.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:18:53.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:52 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:18:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:52 smithi072 ceph-mon[41619]: pgmap v1200: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:18:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:18:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:52 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:18:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:52 smithi173 ceph-mon[33839]: pgmap v1200: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:18:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:18:55.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:54 smithi072 ceph-mon[30177]: pgmap v1201: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:18:55.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:54 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:18:55.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:54 smithi072 ceph-mon[41619]: pgmap v1201: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:18:55.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:54 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:18:55.220 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:18:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:18:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:18:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:54 smithi173 ceph-mon[33839]: pgmap v1201: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:18:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:54 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:18:56.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:55 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:56.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:55 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:18:56.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:55 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:18:56.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:55 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:56.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:55 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:18:56.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:55 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:18:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:55 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:18:56.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:55 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:18:56.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:55 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:18:57.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:57 smithi173 ceph-mon[33839]: pgmap v1202: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:18:57.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:57 smithi072 ceph-mon[30177]: pgmap v1202: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:18:57.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:57 smithi072 ceph-mon[41619]: pgmap v1202: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:18:59.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:18:59 smithi173 ceph-mon[33839]: pgmap v1203: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:18:59.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:18:59 smithi072 ceph-mon[30177]: pgmap v1203: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:18:59.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:18:59 smithi072 ceph-mon[41619]: pgmap v1203: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:01.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:01 smithi173 ceph-mon[33839]: pgmap v1204: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:19:01.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:19:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:19:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:01 smithi072 ceph-mon[30177]: pgmap v1204: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:19:01.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:01 smithi072 ceph-mon[41619]: pgmap v1204: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:19:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:03 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:19:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:03 smithi173 ceph-mon[33839]: pgmap v1205: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:03 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:19:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:03 smithi072 ceph-mon[30177]: pgmap v1205: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:03.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:03 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:19:03.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:03 smithi072 ceph-mon[41619]: pgmap v1205: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:05.234 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:19:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:19:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:19:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:05 smithi173 ceph-mon[33839]: pgmap v1206: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:19:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:05 smithi072 ceph-mon[30177]: pgmap v1206: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:19:05.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:05 smithi072 ceph-mon[41619]: pgmap v1206: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:19:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:07 smithi173 ceph-mon[33839]: pgmap v1207: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:07.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:07 smithi072 ceph-mon[41619]: pgmap v1207: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:07.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:07 smithi072 ceph-mon[30177]: pgmap v1207: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:19:08.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:19:08.752 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:19:09.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:09 smithi173 ceph-mon[33839]: pgmap v1208: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:19:09.722 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:09 smithi072 ceph-mon[41619]: pgmap v1208: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:19:09.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:09 smithi072 ceph-mon[30177]: pgmap v1208: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:19:11.521 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:11 smithi072 ceph-mon[30177]: pgmap v1209: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:19:11.522 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:11 smithi072 ceph-mon[41619]: pgmap v1209: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:19:11.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:11 smithi173 ceph-mon[33839]: pgmap v1209: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:19:11.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:19:11 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:19:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:13 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:19:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:13 smithi173 ceph-mon[33839]: pgmap v1210: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:13.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:13 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:19:13.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:13 smithi072 ceph-mon[30177]: pgmap v1210: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:13.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:13 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:19:13.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:13 smithi072 ceph-mon[41619]: pgmap v1210: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:15.253 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:19:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:19:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:19:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:15 smithi173 ceph-mon[33839]: pgmap v1211: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:15.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:15 smithi072 ceph-mon[30177]: pgmap v1211: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:15.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:15 smithi072 ceph-mon[41619]: pgmap v1211: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:17.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:17 smithi173 ceph-mon[33839]: pgmap v1212: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:17.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:17 smithi072 ceph-mon[30177]: pgmap v1212: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:17.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:17 smithi072 ceph-mon[41619]: pgmap v1212: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:19.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:19 smithi173 ceph-mon[33839]: pgmap v1213: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:19.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:19 smithi072 ceph-mon[30177]: pgmap v1213: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:19.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:19 smithi072 ceph-mon[41619]: pgmap v1213: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:21 smithi173 ceph-mon[33839]: pgmap v1214: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:19:21.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:19:21 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:19:21.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:21 smithi072 ceph-mon[30177]: pgmap v1214: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:19:21.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:21 smithi072 ceph-mon[41619]: pgmap v1214: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:19:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:23 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:19:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:23 smithi173 ceph-mon[33839]: pgmap v1215: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:23 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:19:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:23 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:19:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:23 smithi072 ceph-mon[30177]: pgmap v1215: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:23 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:19:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:23 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:19:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:23 smithi072 ceph-mon[41619]: pgmap v1215: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:23 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:19:25.274 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:19:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:19:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:19:25.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:25 smithi173 ceph-mon[33839]: pgmap v1216: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:25.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:25 smithi072 ceph-mon[30177]: pgmap v1216: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:25.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:25 smithi072 ceph-mon[41619]: pgmap v1216: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:27.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:27 smithi173 ceph-mon[33839]: pgmap v1217: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:27.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:27 smithi072 ceph-mon[30177]: pgmap v1217: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:27.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:27 smithi072 ceph-mon[41619]: pgmap v1217: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:29.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:29 smithi173 ceph-mon[33839]: pgmap v1218: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:29.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:29 smithi072 ceph-mon[30177]: pgmap v1218: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:29.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:29 smithi072 ceph-mon[41619]: pgmap v1218: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:31.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:31 smithi173 ceph-mon[33839]: pgmap v1219: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:19:31.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:19:31 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:19:31.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:31 smithi072 ceph-mon[30177]: pgmap v1219: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:19:31.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:31 smithi072 ceph-mon[41619]: pgmap v1219: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:19:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:33 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:19:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:33 smithi173 ceph-mon[33839]: pgmap v1220: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:33.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:33 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:19:33.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:33 smithi072 ceph-mon[30177]: pgmap v1220: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:33.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:33 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:19:33.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:33 smithi072 ceph-mon[41619]: pgmap v1220: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:35.289 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:19:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:19:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:19:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:35 smithi173 ceph-mon[33839]: pgmap v1221: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:35.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:35 smithi072 ceph-mon[30177]: pgmap v1221: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:35.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:35 smithi072 ceph-mon[41619]: pgmap v1221: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:37.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:36 smithi173 ceph-mon[33839]: pgmap v1222: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:36 smithi072 ceph-mon[30177]: pgmap v1222: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:37.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:36 smithi072 ceph-mon[41619]: pgmap v1222: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:38.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:37 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:19:38.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:37 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:19:38.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:37 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:19:39.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:38 smithi173 ceph-mon[33839]: pgmap v1223: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:39.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:38 smithi072 ceph-mon[30177]: pgmap v1223: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:39.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:38 smithi072 ceph-mon[41619]: pgmap v1223: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:41.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:40 smithi072 ceph-mon[30177]: pgmap v1224: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:19:41.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:40 smithi072 ceph-mon[41619]: pgmap v1224: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:19:41.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:40 smithi173 ceph-mon[33839]: pgmap v1224: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:19:41.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:19:41 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:19:43.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:42 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:19:43.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:42 smithi072 ceph-mon[30177]: pgmap v1225: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:43.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:42 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:19:43.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:42 smithi072 ceph-mon[41619]: pgmap v1225: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:42 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:19:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:42 smithi173 ceph-mon[33839]: pgmap v1225: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:45.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:44 smithi072 ceph-mon[30177]: pgmap v1226: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:45.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:44 smithi072 ceph-mon[41619]: pgmap v1226: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:45.220 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:19:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:19:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:19:45.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:44 smithi173 ceph-mon[33839]: pgmap v1226: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:47.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:46 smithi072 ceph-mon[30177]: pgmap v1227: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:47.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:46 smithi072 ceph-mon[41619]: pgmap v1227: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:47.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:46 smithi173 ceph-mon[33839]: pgmap v1227: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:49.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:48 smithi072 ceph-mon[30177]: pgmap v1228: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:49.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:48 smithi072 ceph-mon[41619]: pgmap v1228: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:48 smithi173 ceph-mon[33839]: pgmap v1228: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:51.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:50 smithi072 ceph-mon[30177]: pgmap v1229: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:19:51.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:50 smithi072 ceph-mon[41619]: pgmap v1229: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:19:51.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:50 smithi173 ceph-mon[33839]: pgmap v1229: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:19:51.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:19:51 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:19:53.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:52 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:19:53.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:52 smithi072 ceph-mon[30177]: pgmap v1230: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:53.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:19:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:52 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:19:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:52 smithi072 ceph-mon[41619]: pgmap v1230: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:19:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:52 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:19:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:52 smithi173 ceph-mon[33839]: pgmap v1230: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:53.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:19:55.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:54 smithi072 ceph-mon[30177]: pgmap v1231: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:55.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:54 smithi072 ceph-mon[41619]: pgmap v1231: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:55.220 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:19:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:19:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:19:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:54 smithi173 ceph-mon[33839]: pgmap v1231: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:56.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:55 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:19:56.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:55 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:19:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:55 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:19:57.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:56 smithi072 ceph-mon[30177]: pgmap v1232: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:57.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:56 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:57.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:56 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:19:57.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:56 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:19:57.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:56 smithi072 ceph-mon[41619]: pgmap v1232: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:57.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:56 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:57.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:56 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:19:57.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:56 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:19:57.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:56 smithi173 ceph-mon[33839]: pgmap v1232: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:19:57.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:56 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:19:57.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:56 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:19:57.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:56 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:19:59.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:19:58 smithi072 ceph-mon[30177]: pgmap v1233: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:59.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:19:58 smithi072 ceph-mon[41619]: pgmap v1233: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:19:59.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:19:58 smithi173 ceph-mon[33839]: pgmap v1233: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:01.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:00 smithi072 ceph-mon[30177]: pgmap v1234: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:20:01.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:00 smithi072 ceph-mon[30177]: overall HEALTH_WARN 4 pool(s) do not have an application enabled 2024-03-26T02:20:01.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:00 smithi072 ceph-mon[41619]: pgmap v1234: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:20:01.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:00 smithi072 ceph-mon[41619]: overall HEALTH_WARN 4 pool(s) do not have an application enabled 2024-03-26T02:20:01.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:00 smithi173 ceph-mon[33839]: pgmap v1234: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:20:01.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:00 smithi173 ceph-mon[33839]: overall HEALTH_WARN 4 pool(s) do not have an application enabled 2024-03-26T02:20:01.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:20:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:20:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:02 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:20:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:02 smithi072 ceph-mon[30177]: pgmap v1235: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:03.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:02 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:20:03.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:02 smithi072 ceph-mon[41619]: pgmap v1235: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:02 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:20:03.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:02 smithi173 ceph-mon[33839]: pgmap v1235: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:05.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:04 smithi072 ceph-mon[30177]: pgmap v1236: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:05.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:04 smithi072 ceph-mon[41619]: pgmap v1236: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:05.220 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:20:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:20:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:20:05.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:04 smithi173 ceph-mon[33839]: pgmap v1236: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:07.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:06 smithi072 ceph-mon[30177]: pgmap v1237: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:07.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:06 smithi072 ceph-mon[41619]: pgmap v1237: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:07.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:06 smithi173 ceph-mon[33839]: pgmap v1237: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:08.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:07 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:20:08.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:07 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:20:08.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:07 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:20:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:08 smithi072 ceph-mon[30177]: pgmap v1238: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:09.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:08 smithi072 ceph-mon[41619]: pgmap v1238: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:08 smithi173 ceph-mon[33839]: pgmap v1238: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:11.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:10 smithi072 ceph-mon[30177]: pgmap v1239: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:20:11.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:10 smithi072 ceph-mon[41619]: pgmap v1239: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:20:11.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:10 smithi173 ceph-mon[33839]: pgmap v1239: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:20:11.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:20:11 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:20:13.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:12 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:20:13.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:12 smithi072 ceph-mon[30177]: pgmap v1240: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:13.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:12 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:20:13.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:12 smithi072 ceph-mon[41619]: pgmap v1240: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:12 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:20:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:12 smithi173 ceph-mon[33839]: pgmap v1240: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:15.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:14 smithi072 ceph-mon[30177]: pgmap v1241: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:20:15.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:14 smithi072 ceph-mon[41619]: pgmap v1241: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:20:15.220 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:20:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:20:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:20:15.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:14 smithi173 ceph-mon[33839]: pgmap v1241: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:20:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:16 smithi072 ceph-mon[30177]: pgmap v1242: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:17.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:16 smithi072 ceph-mon[41619]: pgmap v1242: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:17.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:16 smithi173 ceph-mon[33839]: pgmap v1242: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:19.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:18 smithi072 ceph-mon[30177]: pgmap v1243: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:19.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:18 smithi072 ceph-mon[41619]: pgmap v1243: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:19.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:18 smithi173 ceph-mon[33839]: pgmap v1243: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:21.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:20 smithi072 ceph-mon[30177]: pgmap v1244: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:20:21.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:20 smithi072 ceph-mon[41619]: pgmap v1244: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:20:21.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:20 smithi173 ceph-mon[33839]: pgmap v1244: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:20:21.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:20:21 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:20:23.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:22 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:20:23.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:22 smithi072 ceph-mon[41619]: pgmap v1245: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:23.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:22 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:20:23.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:22 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:20:23.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:22 smithi072 ceph-mon[30177]: pgmap v1245: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:23.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:22 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:20:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:22 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:20:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:22 smithi173 ceph-mon[33839]: pgmap v1245: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:22 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:20:25.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:24 smithi072 ceph-mon[30177]: pgmap v1246: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:25.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:24 smithi072 ceph-mon[41619]: pgmap v1246: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:25.220 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:20:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:20:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:20:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:24 smithi173 ceph-mon[33839]: pgmap v1246: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:26 smithi072 ceph-mon[30177]: pgmap v1247: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:27.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:26 smithi072 ceph-mon[41619]: pgmap v1247: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:27.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:26 smithi173 ceph-mon[33839]: pgmap v1247: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:29.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:28 smithi072 ceph-mon[30177]: pgmap v1248: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:29.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:28 smithi072 ceph-mon[41619]: pgmap v1248: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:28 smithi173 ceph-mon[33839]: pgmap v1248: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:31.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:30 smithi072 ceph-mon[30177]: pgmap v1249: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:20:31.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:30 smithi072 ceph-mon[41619]: pgmap v1249: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:20:31.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:30 smithi173 ceph-mon[33839]: pgmap v1249: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:20:31.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:20:31 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:20:33.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:32 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:20:33.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:32 smithi072 ceph-mon[30177]: pgmap v1250: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:33.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:32 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:20:33.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:32 smithi072 ceph-mon[41619]: pgmap v1250: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:33.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:32 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:20:33.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:32 smithi173 ceph-mon[33839]: pgmap v1250: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:35.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:34 smithi072 ceph-mon[30177]: pgmap v1251: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:35.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:34 smithi072 ceph-mon[41619]: pgmap v1251: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:35.220 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:20:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:20:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:20:35.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:34 smithi173 ceph-mon[33839]: pgmap v1251: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:36 smithi072 ceph-mon[30177]: pgmap v1252: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:37.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:36 smithi072 ceph-mon[41619]: pgmap v1252: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:36 smithi173 ceph-mon[33839]: pgmap v1252: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:38.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:37 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:20:38.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:37 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:20:38.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:37 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:20:39.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:38 smithi173 ceph-mon[33839]: pgmap v1253: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:39.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:38 smithi072 ceph-mon[30177]: pgmap v1253: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:39.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:38 smithi072 ceph-mon[41619]: pgmap v1253: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:41.271 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:40 smithi072 ceph-mon[30177]: pgmap v1254: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:20:41.272 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:40 smithi072 ceph-mon[41619]: pgmap v1254: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:20:41.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:40 smithi173 ceph-mon[33839]: pgmap v1254: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:20:41.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:20:41 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:20:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:42 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:20:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:42 smithi173 ceph-mon[33839]: pgmap v1255: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:42 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:20:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:42 smithi072 ceph-mon[30177]: pgmap v1255: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:43.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:42 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:20:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:42 smithi072 ceph-mon[41619]: pgmap v1255: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:45.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:44 smithi173 ceph-mon[33839]: pgmap v1256: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:45.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:44 smithi072 ceph-mon[30177]: pgmap v1256: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:45.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:20:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:20:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:20:45.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:44 smithi072 ceph-mon[41619]: pgmap v1256: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:47.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:46 smithi173 ceph-mon[33839]: pgmap v1257: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:47.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:46 smithi072 ceph-mon[30177]: pgmap v1257: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:47.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:46 smithi072 ceph-mon[41619]: pgmap v1257: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:48 smithi173 ceph-mon[33839]: pgmap v1258: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:48 smithi072 ceph-mon[30177]: pgmap v1258: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:49.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:48 smithi072 ceph-mon[41619]: pgmap v1258: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:51.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:50 smithi173 ceph-mon[33839]: pgmap v1259: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:20:51.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:50 smithi072 ceph-mon[41619]: pgmap v1259: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:20:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:50 smithi072 ceph-mon[30177]: pgmap v1259: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:20:51.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:20:51 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:20:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:52 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:20:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:52 smithi173 ceph-mon[33839]: pgmap v1260: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:20:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:52 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:20:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:52 smithi072 ceph-mon[30177]: pgmap v1260: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:20:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:52 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:20:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:52 smithi072 ceph-mon[41619]: pgmap v1260: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:20:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:54 smithi173 ceph-mon[33839]: pgmap v1261: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:55.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:54 smithi072 ceph-mon[30177]: pgmap v1261: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:55.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:54 smithi072 ceph-mon[41619]: pgmap v1261: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:55.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:20:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:20:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:20:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:55 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:20:56.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:55 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:20:56.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:56 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:20:57.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:57 smithi173 ceph-mon[33839]: pgmap v1262: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:57.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:57 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:57.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:57 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:20:57.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:57 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:20:57.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:57 smithi072 ceph-mon[30177]: pgmap v1262: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:57.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:57 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:57.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:57 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:20:57.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:57 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:20:57.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:57 smithi072 ceph-mon[41619]: pgmap v1262: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:20:57.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:57 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:20:57.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:57 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:20:57.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:57 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:20:59.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:20:59 smithi173 ceph-mon[33839]: pgmap v1263: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:59.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:20:59 smithi072 ceph-mon[30177]: pgmap v1263: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:20:59.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:20:59 smithi072 ceph-mon[41619]: pgmap v1263: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:01.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:01 smithi173 ceph-mon[33839]: pgmap v1264: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:21:01.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:01 smithi072 ceph-mon[30177]: pgmap v1264: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:21:01.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:01 smithi072 ceph-mon[41619]: pgmap v1264: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:21:01.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:21:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:21:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:03 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:21:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:03 smithi173 ceph-mon[33839]: pgmap v1265: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:21:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:03 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:21:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:03 smithi072 ceph-mon[30177]: pgmap v1265: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:21:03.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:03 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:21:03.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:03 smithi072 ceph-mon[41619]: pgmap v1265: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:21:05.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:05 smithi173 ceph-mon[33839]: pgmap v1266: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:05.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:05 smithi072 ceph-mon[30177]: pgmap v1266: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:05.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:05 smithi072 ceph-mon[41619]: pgmap v1266: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:05.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:21:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:21:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:21:07.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:07 smithi173 ceph-mon[33839]: pgmap v1267: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:21:07.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:07 smithi072 ceph-mon[30177]: pgmap v1267: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:21:07.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:07 smithi072 ceph-mon[41619]: pgmap v1267: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:21:08.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:21:08.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:21:08.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:21:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:09 smithi173 ceph-mon[33839]: pgmap v1268: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:09.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:09 smithi072 ceph-mon[30177]: pgmap v1268: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:09.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:09 smithi072 ceph-mon[41619]: pgmap v1268: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:11.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:11 smithi173 ceph-mon[33839]: pgmap v1269: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:21:11.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:11 smithi072 ceph-mon[30177]: pgmap v1269: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:21:11.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:11 smithi072 ceph-mon[41619]: pgmap v1269: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:21:11.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:21:11 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:21:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:13 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:21:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:13 smithi173 ceph-mon[33839]: pgmap v1270: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:21:13.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:13 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:21:13.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:13 smithi072 ceph-mon[30177]: pgmap v1270: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:21:13.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:13 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:21:13.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:13 smithi072 ceph-mon[41619]: pgmap v1270: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:21:15.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:15 smithi173 ceph-mon[33839]: pgmap v1271: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:15.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:15 smithi072 ceph-mon[30177]: pgmap v1271: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:15.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:15 smithi072 ceph-mon[41619]: pgmap v1271: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:15.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:21:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:21:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:21:17.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:17 smithi173 ceph-mon[33839]: pgmap v1272: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T02:21:17.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:17 smithi072 ceph-mon[30177]: pgmap v1272: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T02:21:17.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:17 smithi072 ceph-mon[41619]: pgmap v1272: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T02:21:19.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:19 smithi173 ceph-mon[33839]: pgmap v1273: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T02:21:19.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:19 smithi072 ceph-mon[30177]: pgmap v1273: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T02:21:19.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:19 smithi072 ceph-mon[41619]: pgmap v1273: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T02:21:21.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:21 smithi173 ceph-mon[33839]: pgmap v1274: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T02:21:21.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:21 smithi072 ceph-mon[30177]: pgmap v1274: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T02:21:21.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:21 smithi072 ceph-mon[41619]: pgmap v1274: 260 pgs: 260 active+clean; 582 KiB data, 742 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T02:21:21.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:21:21 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:21:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:23 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:21:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:23 smithi173 ceph-mon[33839]: pgmap v1275: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T02:21:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:23 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:21:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:23 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:21:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:23 smithi072 ceph-mon[30177]: pgmap v1275: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T02:21:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:23 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:21:23.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:23 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:21:23.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:23 smithi072 ceph-mon[41619]: pgmap v1275: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T02:21:23.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:23 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:21:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:25 smithi173 ceph-mon[33839]: pgmap v1276: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T02:21:25.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:25 smithi072 ceph-mon[30177]: pgmap v1276: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T02:21:25.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:25 smithi072 ceph-mon[41619]: pgmap v1276: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T02:21:25.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:21:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:21:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:21:27.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:27 smithi173 ceph-mon[33839]: pgmap v1277: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T02:21:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:27 smithi072 ceph-mon[30177]: pgmap v1277: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T02:21:27.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:27 smithi072 ceph-mon[41619]: pgmap v1277: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T02:21:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:29 smithi173 ceph-mon[33839]: pgmap v1278: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-03-26T02:21:29.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:29 smithi072 ceph-mon[30177]: pgmap v1278: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-03-26T02:21:29.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:29 smithi072 ceph-mon[41619]: pgmap v1278: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-03-26T02:21:31.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:31 smithi173 ceph-mon[33839]: pgmap v1279: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-03-26T02:21:31.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:31 smithi072 ceph-mon[30177]: pgmap v1279: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-03-26T02:21:31.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:31 smithi072 ceph-mon[41619]: pgmap v1279: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-03-26T02:21:31.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:21:31 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:21:33.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:33 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:21:33.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:33 smithi072 ceph-mon[41619]: pgmap v1280: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-03-26T02:21:33.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:33 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:21:33.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:33 smithi072 ceph-mon[30177]: pgmap v1280: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-03-26T02:21:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:33 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:21:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:33 smithi173 ceph-mon[33839]: pgmap v1280: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-03-26T02:21:35.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:35 smithi072 ceph-mon[30177]: pgmap v1281: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:35.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:35 smithi072 ceph-mon[41619]: pgmap v1281: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:35.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:21:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:21:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:21:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:35 smithi173 ceph-mon[33839]: pgmap v1281: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:37.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:37 smithi072 ceph-mon[30177]: pgmap v1282: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:21:37.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:37 smithi072 ceph-mon[41619]: pgmap v1282: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:21:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:37 smithi173 ceph-mon[33839]: pgmap v1282: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:21:38.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:38 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:21:38.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:38 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:21:38.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:38 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:21:39.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:39 smithi072 ceph-mon[30177]: pgmap v1283: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:39.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:39 smithi072 ceph-mon[41619]: pgmap v1283: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:39.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:39 smithi173 ceph-mon[33839]: pgmap v1283: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:41.365 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:41 smithi173 ceph-mon[33839]: pgmap v1284: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:21:41.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:41 smithi072 ceph-mon[30177]: pgmap v1284: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:21:41.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:41 smithi072 ceph-mon[41619]: pgmap v1284: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:21:41.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:21:41 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:21:43.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:43 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:21:43.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:43 smithi072 ceph-mon[41619]: pgmap v1285: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:21:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:43 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:21:43.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:43 smithi072 ceph-mon[30177]: pgmap v1285: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:21:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:43 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:21:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:43 smithi173 ceph-mon[33839]: pgmap v1285: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:21:45.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:45 smithi072 ceph-mon[30177]: pgmap v1286: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:45.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:45 smithi072 ceph-mon[41619]: pgmap v1286: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:45.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:21:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:21:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:21:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:45 smithi173 ceph-mon[33839]: pgmap v1286: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:47.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:47 smithi072 ceph-mon[30177]: pgmap v1287: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:21:47.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:47 smithi072 ceph-mon[41619]: pgmap v1287: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:21:47.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:47 smithi173 ceph-mon[33839]: pgmap v1287: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:21:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:49 smithi072 ceph-mon[30177]: pgmap v1288: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:49.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:49 smithi072 ceph-mon[41619]: pgmap v1288: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:49.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:49 smithi173 ceph-mon[33839]: pgmap v1288: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:51 smithi072 ceph-mon[30177]: pgmap v1289: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:51.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:51 smithi072 ceph-mon[41619]: pgmap v1289: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:51.516 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:51 smithi173 ceph-mon[33839]: pgmap v1289: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:51.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:21:51 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:21:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:53 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:21:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:53 smithi072 ceph-mon[30177]: pgmap v1290: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:21:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:53 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:21:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:53 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:21:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:53 smithi072 ceph-mon[41619]: pgmap v1290: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:21:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:53 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:21:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:53 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:21:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:53 smithi173 ceph-mon[33839]: pgmap v1290: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:21:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:53 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:21:55.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:55 smithi072 ceph-mon[30177]: pgmap v1291: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:55.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:55 smithi072 ceph-mon[41619]: pgmap v1291: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:55.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:21:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:21:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:21:55.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:55 smithi173 ceph-mon[33839]: pgmap v1291: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:57.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:57 smithi072 ceph-mon[30177]: pgmap v1292: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:21:57.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:57 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:21:57.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:57 smithi072 ceph-mon[41619]: pgmap v1292: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:21:57.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:57 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:21:57.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:57 smithi173 ceph-mon[33839]: pgmap v1292: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:21:57.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:57 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:21:58.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:58 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:58.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:58 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:21:58.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:58 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:21:58.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:58 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:58.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:58 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:21:58.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:58 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:21:58.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:58 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:21:58.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:58 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:21:58.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:58 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:21:59.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:21:59 smithi072 ceph-mon[30177]: pgmap v1293: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:59.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:21:59 smithi072 ceph-mon[41619]: pgmap v1293: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:21:59.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:21:59 smithi173 ceph-mon[33839]: pgmap v1293: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:01 smithi072 ceph-mon[30177]: pgmap v1294: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:22:01.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:01 smithi072 ceph-mon[41619]: pgmap v1294: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:22:01.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:01 smithi173 ceph-mon[33839]: pgmap v1294: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:22:01.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:22:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:22:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:03 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:22:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:03 smithi072 ceph-mon[30177]: pgmap v1295: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:03.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:03 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:22:03.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:03 smithi072 ceph-mon[41619]: pgmap v1295: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:03 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:22:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:03 smithi173 ceph-mon[33839]: pgmap v1295: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:05.322 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:22:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:22:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:22:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:05 smithi072 ceph-mon[30177]: pgmap v1296: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:05.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:05 smithi072 ceph-mon[41619]: pgmap v1296: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:05.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:05 smithi173 ceph-mon[33839]: pgmap v1296: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:07.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:07 smithi072 ceph-mon[30177]: pgmap v1297: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:07.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:07 smithi072 ceph-mon[41619]: pgmap v1297: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:07 smithi173 ceph-mon[33839]: pgmap v1297: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:08.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:22:08.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:22:08.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:22:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:09 smithi072 ceph-mon[30177]: pgmap v1298: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:09.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:09 smithi072 ceph-mon[41619]: pgmap v1298: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:09.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:09 smithi173 ceph-mon[33839]: pgmap v1298: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:11.616 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:11 smithi173 ceph-mon[33839]: pgmap v1299: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:22:11.616 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:22:11 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:22:11.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:11 smithi072 ceph-mon[30177]: pgmap v1299: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:22:11.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:11 smithi072 ceph-mon[41619]: pgmap v1299: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:22:13.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:13 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:22:13.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:13 smithi072 ceph-mon[41619]: pgmap v1300: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:13.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:13 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:22:13.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:13 smithi072 ceph-mon[30177]: pgmap v1300: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:13.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:13 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:22:13.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:13 smithi173 ceph-mon[33839]: pgmap v1300: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:15.342 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:22:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:22:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:22:15.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:15 smithi072 ceph-mon[30177]: pgmap v1301: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:15.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:15 smithi072 ceph-mon[41619]: pgmap v1301: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:15.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:15 smithi173 ceph-mon[33839]: pgmap v1301: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:16 smithi072 ceph-mon[30177]: pgmap v1302: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:17.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:16 smithi072 ceph-mon[41619]: pgmap v1302: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:17.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:16 smithi173 ceph-mon[33839]: pgmap v1302: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:19.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:18 smithi072 ceph-mon[30177]: pgmap v1303: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:19.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:18 smithi072 ceph-mon[41619]: pgmap v1303: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:19.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:18 smithi173 ceph-mon[33839]: pgmap v1303: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:21.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:20 smithi072 ceph-mon[30177]: pgmap v1304: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:21.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:20 smithi072 ceph-mon[41619]: pgmap v1304: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:21.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:20 smithi173 ceph-mon[33839]: pgmap v1304: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:21.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:22:21 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:22:23.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:22 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:22:23.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:22 smithi072 ceph-mon[30177]: pgmap v1305: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:23.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:22 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:22:23.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:22 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:22:23.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:22 smithi072 ceph-mon[41619]: pgmap v1305: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:23.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:22 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:22:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:22 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:22:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:22 smithi173 ceph-mon[33839]: pgmap v1305: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:22 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:22:25.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:24 smithi072 ceph-mon[30177]: pgmap v1306: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:25.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:24 smithi072 ceph-mon[41619]: pgmap v1306: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:25.220 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:22:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:22:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:22:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:24 smithi173 ceph-mon[33839]: pgmap v1306: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:26 smithi072 ceph-mon[30177]: pgmap v1307: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:27.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:26 smithi072 ceph-mon[41619]: pgmap v1307: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:27.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:26 smithi173 ceph-mon[33839]: pgmap v1307: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:29.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:28 smithi072 ceph-mon[30177]: pgmap v1308: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:29.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:28 smithi072 ceph-mon[41619]: pgmap v1308: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:28 smithi173 ceph-mon[33839]: pgmap v1308: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:31.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:30 smithi072 ceph-mon[30177]: pgmap v1309: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:31.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:30 smithi072 ceph-mon[41619]: pgmap v1309: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:31.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:30 smithi173 ceph-mon[33839]: pgmap v1309: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:31.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:22:31 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:22:33.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:32 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:22:33.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:32 smithi072 ceph-mon[41619]: pgmap v1310: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:33.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:32 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:22:33.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:32 smithi072 ceph-mon[30177]: pgmap v1310: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:33.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:32 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:22:33.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:32 smithi173 ceph-mon[33839]: pgmap v1310: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:35.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:34 smithi072 ceph-mon[30177]: pgmap v1311: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:35.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:34 smithi072 ceph-mon[41619]: pgmap v1311: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:35.220 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:22:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:22:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:22:35.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:34 smithi173 ceph-mon[33839]: pgmap v1311: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:36 smithi072 ceph-mon[30177]: pgmap v1312: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:37.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:36 smithi072 ceph-mon[41619]: pgmap v1312: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:36 smithi173 ceph-mon[33839]: pgmap v1312: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:38.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:37 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:22:38.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:37 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:22:38.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:37 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:22:39.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:38 smithi072 ceph-mon[30177]: pgmap v1313: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:39.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:38 smithi072 ceph-mon[41619]: pgmap v1313: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:39.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:38 smithi173 ceph-mon[33839]: pgmap v1313: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:41.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:40 smithi072 ceph-mon[30177]: pgmap v1314: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:22:41.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:40 smithi072 ceph-mon[41619]: pgmap v1314: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:22:41.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:40 smithi173 ceph-mon[33839]: pgmap v1314: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:22:42.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:22:41 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:22:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:42 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:22:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:42 smithi173 ceph-mon[33839]: pgmap v1315: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:42 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:22:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:42 smithi072 ceph-mon[30177]: pgmap v1315: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:42 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:22:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:42 smithi072 ceph-mon[41619]: pgmap v1315: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:45.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:44 smithi173 ceph-mon[33839]: pgmap v1316: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:45.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:44 smithi072 ceph-mon[30177]: pgmap v1316: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:45.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:22:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:22:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:22:45.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:44 smithi072 ceph-mon[41619]: pgmap v1316: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:47.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:46 smithi173 ceph-mon[33839]: pgmap v1317: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:47.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:46 smithi072 ceph-mon[30177]: pgmap v1317: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:47.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:46 smithi072 ceph-mon[41619]: pgmap v1317: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:48 smithi173 ceph-mon[33839]: pgmap v1318: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:48 smithi072 ceph-mon[30177]: pgmap v1318: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:49.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:48 smithi072 ceph-mon[41619]: pgmap v1318: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:51.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:50 smithi173 ceph-mon[33839]: pgmap v1319: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:22:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:50 smithi072 ceph-mon[30177]: pgmap v1319: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:22:51.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:50 smithi072 ceph-mon[41619]: pgmap v1319: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:22:52.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:22:51 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:22:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:52 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:22:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:52 smithi173 ceph-mon[33839]: pgmap v1320: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:22:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:52 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:22:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:52 smithi072 ceph-mon[30177]: pgmap v1320: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:22:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:52 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:22:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:52 smithi072 ceph-mon[41619]: pgmap v1320: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:22:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:54 smithi173 ceph-mon[33839]: pgmap v1321: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:55.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:54 smithi072 ceph-mon[30177]: pgmap v1321: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:55.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:54 smithi072 ceph-mon[41619]: pgmap v1321: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:55.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:22:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:22:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:22:56.994 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:22:56 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=cleanup t=2024-03-26T02:22:56.640036018Z level=info msg="Completed cleanup jobs" duration=6.680711ms 2024-03-26T02:22:57.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:56 smithi173 ceph-mon[33839]: pgmap v1322: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:57.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:56 smithi072 ceph-mon[30177]: pgmap v1322: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:57.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:56 smithi072 ceph-mon[41619]: pgmap v1322: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:22:58.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:57 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:22:58.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:57 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:22:58.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:57 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:22:59.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:59 smithi173 ceph-mon[33839]: pgmap v1323: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:59.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:59 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:22:59.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:59 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:22:59.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:22:59 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:22:59.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:59 smithi072 ceph-mon[30177]: pgmap v1323: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:59.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:59 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:22:59.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:59 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:22:59.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:22:59 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:22:59.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:59 smithi072 ceph-mon[41619]: pgmap v1323: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:22:59.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:59 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:22:59.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:59 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:22:59.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:22:59 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:23:01.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:01 smithi173 ceph-mon[33839]: pgmap v1324: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:23:01.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:01 smithi072 ceph-mon[30177]: pgmap v1324: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:23:01.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:01 smithi072 ceph-mon[41619]: pgmap v1324: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:23:02.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:23:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:23:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:03 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:23:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:03 smithi173 ceph-mon[33839]: pgmap v1325: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:03 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:23:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:03 smithi072 ceph-mon[30177]: pgmap v1325: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:03.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:03 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:23:03.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:03 smithi072 ceph-mon[41619]: pgmap v1325: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:05.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:05 smithi173 ceph-mon[33839]: pgmap v1326: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:23:05.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:05 smithi072 ceph-mon[30177]: pgmap v1326: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:23:05.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:05 smithi072 ceph-mon[41619]: pgmap v1326: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:23:05.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:23:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:23:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:23:07.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:07 smithi173 ceph-mon[33839]: pgmap v1327: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:07.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:07 smithi072 ceph-mon[30177]: pgmap v1327: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:07.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:07 smithi072 ceph-mon[41619]: pgmap v1327: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:08.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:23:08.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:23:08.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:23:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:09 smithi173 ceph-mon[33839]: pgmap v1328: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:23:09.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:09 smithi072 ceph-mon[30177]: pgmap v1328: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:23:09.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:09 smithi072 ceph-mon[41619]: pgmap v1328: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:23:11.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:11 smithi173 ceph-mon[33839]: pgmap v1329: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:23:11.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:11 smithi072 ceph-mon[30177]: pgmap v1329: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:23:11.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:11 smithi072 ceph-mon[41619]: pgmap v1329: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:23:12.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:23:11 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:23:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:13 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:23:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:13 smithi173 ceph-mon[33839]: pgmap v1330: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:13.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:13 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:23:13.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:13 smithi072 ceph-mon[30177]: pgmap v1330: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:13.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:13 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:23:13.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:13 smithi072 ceph-mon[41619]: pgmap v1330: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:15.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:15 smithi173 ceph-mon[33839]: pgmap v1331: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:15.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:15 smithi072 ceph-mon[30177]: pgmap v1331: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:15.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:23:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:23:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:23:15.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:15 smithi072 ceph-mon[41619]: pgmap v1331: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:17.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:16 smithi072 ceph-mon[30177]: pgmap v1332: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:17.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:16 smithi072 ceph-mon[41619]: pgmap v1332: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:17.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:16 smithi173 ceph-mon[33839]: pgmap v1332: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:19.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:18 smithi072 ceph-mon[30177]: pgmap v1333: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:19.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:18 smithi072 ceph-mon[41619]: pgmap v1333: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:19.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:18 smithi173 ceph-mon[33839]: pgmap v1333: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:21.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:20 smithi072 ceph-mon[30177]: pgmap v1334: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:21.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:20 smithi072 ceph-mon[41619]: pgmap v1334: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:21.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:20 smithi173 ceph-mon[33839]: pgmap v1334: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:22.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:23:21 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:23:23.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:22 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:23:23.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:22 smithi072 ceph-mon[41619]: pgmap v1335: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:23.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:22 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:23:23.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:22 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:23:23.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:22 smithi072 ceph-mon[30177]: pgmap v1335: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:23.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:22 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:23:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:22 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:23:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:22 smithi173 ceph-mon[33839]: pgmap v1335: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:22 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:23:25.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:24 smithi072 ceph-mon[30177]: pgmap v1336: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:25.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:24 smithi072 ceph-mon[41619]: pgmap v1336: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:25.220 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:23:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:23:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:23:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:24 smithi173 ceph-mon[33839]: pgmap v1336: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:27.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:26 smithi072 ceph-mon[30177]: pgmap v1337: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:27.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:26 smithi072 ceph-mon[41619]: pgmap v1337: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:27.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:26 smithi173 ceph-mon[33839]: pgmap v1337: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:29.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:28 smithi072 ceph-mon[30177]: pgmap v1338: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:29.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:28 smithi072 ceph-mon[41619]: pgmap v1338: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:28 smithi173 ceph-mon[33839]: pgmap v1338: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:31.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:30 smithi072 ceph-mon[30177]: pgmap v1339: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:23:31.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:30 smithi072 ceph-mon[41619]: pgmap v1339: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:23:31.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:30 smithi173 ceph-mon[33839]: pgmap v1339: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:23:32.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:23:31 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:23:33.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:32 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:23:33.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:32 smithi173 ceph-mon[33839]: pgmap v1340: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:33.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:32 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:23:33.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:32 smithi072 ceph-mon[30177]: pgmap v1340: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:33.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:32 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:23:33.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:32 smithi072 ceph-mon[41619]: pgmap v1340: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:35.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:34 smithi173 ceph-mon[33839]: pgmap v1341: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:35.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:34 smithi072 ceph-mon[30177]: pgmap v1341: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:35.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:23:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:23:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:23:35.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:34 smithi072 ceph-mon[41619]: pgmap v1341: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:36 smithi072 ceph-mon[30177]: pgmap v1342: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:37.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:36 smithi072 ceph-mon[41619]: pgmap v1342: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:36 smithi173 ceph-mon[33839]: pgmap v1342: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:38.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:37 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:23:38.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:37 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:23:38.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:37 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:23:39.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:38 smithi072 ceph-mon[30177]: pgmap v1343: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:39.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:38 smithi072 ceph-mon[41619]: pgmap v1343: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:39.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:38 smithi173 ceph-mon[33839]: pgmap v1343: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:41.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:40 smithi072 ceph-mon[30177]: pgmap v1344: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:41.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:40 smithi072 ceph-mon[41619]: pgmap v1344: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:41.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:40 smithi173 ceph-mon[33839]: pgmap v1344: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:42.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:23:41 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:23:43.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:42 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:23:43.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:42 smithi072 ceph-mon[30177]: pgmap v1345: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:43.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:42 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:23:43.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:42 smithi072 ceph-mon[41619]: pgmap v1345: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:42 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:23:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:42 smithi173 ceph-mon[33839]: pgmap v1345: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:45.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:44 smithi072 ceph-mon[30177]: pgmap v1346: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:45.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:44 smithi072 ceph-mon[41619]: pgmap v1346: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:45.220 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:23:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:23:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:23:45.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:44 smithi173 ceph-mon[33839]: pgmap v1346: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:47.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:46 smithi072 ceph-mon[30177]: pgmap v1347: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:47.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:46 smithi072 ceph-mon[41619]: pgmap v1347: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:47.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:46 smithi173 ceph-mon[33839]: pgmap v1347: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:49.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:48 smithi173 ceph-mon[33839]: pgmap v1348: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:48 smithi072 ceph-mon[30177]: pgmap v1348: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:49.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:48 smithi072 ceph-mon[41619]: pgmap v1348: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:51.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:50 smithi173 ceph-mon[33839]: pgmap v1349: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:23:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:50 smithi072 ceph-mon[30177]: pgmap v1349: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:23:51.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:50 smithi072 ceph-mon[41619]: pgmap v1349: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:23:52.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:23:51 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:23:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:52 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:23:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:52 smithi173 ceph-mon[33839]: pgmap v1350: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:52 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:23:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:52 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:23:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:52 smithi072 ceph-mon[30177]: pgmap v1350: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:52 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:23:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:52 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:23:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:52 smithi072 ceph-mon[41619]: pgmap v1350: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:52 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:23:55.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:54 smithi173 ceph-mon[33839]: pgmap v1351: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:55.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:54 smithi072 ceph-mon[30177]: pgmap v1351: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:55.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:54 smithi072 ceph-mon[41619]: pgmap v1351: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:55.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:23:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:23:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:23:57.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:56 smithi173 ceph-mon[33839]: pgmap v1352: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:57.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:56 smithi072 ceph-mon[30177]: pgmap v1352: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:57.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:56 smithi072 ceph-mon[41619]: pgmap v1352: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:23:59.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:58 smithi173 ceph-mon[33839]: pgmap v1353: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:59.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:58 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:23:59.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:58 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:23:59.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:58 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:23:59.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:23:58 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:23:59.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:58 smithi072 ceph-mon[30177]: pgmap v1353: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:59.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:58 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:23:59.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:58 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:23:59.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:58 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:23:59.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:23:58 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:23:59.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:58 smithi072 ceph-mon[41619]: pgmap v1353: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:23:59.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:58 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:23:59.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:58 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:23:59.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:58 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:23:59.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:23:58 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:24:01.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:00 smithi173 ceph-mon[33839]: pgmap v1354: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:24:01.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:00 smithi072 ceph-mon[30177]: pgmap v1354: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:24:01.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:00 smithi072 ceph-mon[41619]: pgmap v1354: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:24:02.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:24:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:24:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:02 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:03.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:02 smithi173 ceph-mon[33839]: pgmap v1355: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:02 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:03.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:02 smithi072 ceph-mon[30177]: pgmap v1355: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:03.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:02 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:03.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:02 smithi072 ceph-mon[41619]: pgmap v1355: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:05.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:04 smithi173 ceph-mon[33839]: pgmap v1356: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:05.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:04 smithi072 ceph-mon[30177]: pgmap v1356: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:05.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:24:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:24:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:24:05.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:04 smithi072 ceph-mon[41619]: pgmap v1356: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:07.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:06 smithi173 ceph-mon[33839]: pgmap v1357: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:07.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:06 smithi072 ceph-mon[30177]: pgmap v1357: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:07.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:06 smithi072 ceph-mon[41619]: pgmap v1357: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:08.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:24:08.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:24:08.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:24:09.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:09 smithi173 ceph-mon[33839]: pgmap v1358: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:09.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:09 smithi072 ceph-mon[30177]: pgmap v1358: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:09.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:09 smithi072 ceph-mon[41619]: pgmap v1358: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:11.271 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:11 smithi072 ceph-mon[30177]: pgmap v1359: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:11.271 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:11 smithi072 ceph-mon[41619]: pgmap v1359: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:11.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:11 smithi173 ceph-mon[33839]: pgmap v1359: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:12.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:24:11 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:24:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:13 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:13 smithi173 ceph-mon[33839]: pgmap v1360: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:13.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:13 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:13.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:13 smithi072 ceph-mon[41619]: pgmap v1360: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:13.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:13 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:13.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:13 smithi072 ceph-mon[30177]: pgmap v1360: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:15.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:15 smithi173 ceph-mon[33839]: pgmap v1361: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:15.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:15 smithi072 ceph-mon[30177]: pgmap v1361: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:15.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:15 smithi072 ceph-mon[41619]: pgmap v1361: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:15.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:24:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:24:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:24:17.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:17 smithi173 ceph-mon[33839]: pgmap v1362: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:17.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:17 smithi072 ceph-mon[30177]: pgmap v1362: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:17.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:17 smithi072 ceph-mon[41619]: pgmap v1362: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:19.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:19 smithi173 ceph-mon[33839]: pgmap v1363: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:19.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:19 smithi072 ceph-mon[30177]: pgmap v1363: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:19.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:19 smithi072 ceph-mon[41619]: pgmap v1363: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:21.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:21 smithi173 ceph-mon[33839]: pgmap v1364: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:24:21.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:21 smithi072 ceph-mon[30177]: pgmap v1364: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:24:21.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:21 smithi072 ceph-mon[41619]: pgmap v1364: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:24:22.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:24:21 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:24:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:23 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:23 smithi173 ceph-mon[33839]: pgmap v1365: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:23 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:24:23.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:23 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:23.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:23 smithi072 ceph-mon[41619]: pgmap v1365: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:23.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:23 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:24:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:23 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:23 smithi072 ceph-mon[30177]: pgmap v1365: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:23.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:23 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:24:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:25 smithi173 ceph-mon[33839]: pgmap v1366: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:25.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:25 smithi072 ceph-mon[30177]: pgmap v1366: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:25.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:25 smithi072 ceph-mon[41619]: pgmap v1366: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:25.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:24:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:24:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:24:27.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:27 smithi173 ceph-mon[33839]: pgmap v1367: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:27 smithi072 ceph-mon[30177]: pgmap v1367: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:27.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:27 smithi072 ceph-mon[41619]: pgmap v1367: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:29 smithi173 ceph-mon[33839]: pgmap v1368: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:29.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:29 smithi072 ceph-mon[30177]: pgmap v1368: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:29.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:29 smithi072 ceph-mon[41619]: pgmap v1368: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:31.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:31 smithi173 ceph-mon[33839]: pgmap v1369: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:24:31.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:31 smithi072 ceph-mon[30177]: pgmap v1369: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:24:31.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:31 smithi072 ceph-mon[41619]: pgmap v1369: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:24:32.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:24:31 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:24:33.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:33 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:33.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:33 smithi173 ceph-mon[33839]: pgmap v1370: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:33.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:33 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:33.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:33 smithi072 ceph-mon[30177]: pgmap v1370: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:33.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:33 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:33.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:33 smithi072 ceph-mon[41619]: pgmap v1370: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:35.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:35 smithi173 ceph-mon[33839]: pgmap v1371: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:35.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:35 smithi072 ceph-mon[30177]: pgmap v1371: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:35.471 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:24:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:24:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:24:35.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:35 smithi072 ceph-mon[41619]: pgmap v1371: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:37 smithi173 ceph-mon[33839]: pgmap v1372: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:37.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:37 smithi072 ceph-mon[30177]: pgmap v1372: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:37.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:37 smithi072 ceph-mon[41619]: pgmap v1372: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:38.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:38 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:24:38.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:38 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:24:38.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:38 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:24:39.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:39 smithi173 ceph-mon[33839]: pgmap v1373: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:39.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:39 smithi072 ceph-mon[30177]: pgmap v1373: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:39.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:39 smithi072 ceph-mon[41619]: pgmap v1373: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:41.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:41 smithi173 ceph-mon[33839]: pgmap v1374: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:24:41.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:41 smithi072 ceph-mon[30177]: pgmap v1374: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:24:41.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:41 smithi072 ceph-mon[41619]: pgmap v1374: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:24:42.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:24:41 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:24:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:43 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:43 smithi072 ceph-mon[30177]: pgmap v1375: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:43 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:43 smithi072 ceph-mon[41619]: pgmap v1375: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:43 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:43 smithi173 ceph-mon[33839]: pgmap v1375: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:45.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:45 smithi072 ceph-mon[30177]: pgmap v1376: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:45.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:24:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:24:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:24:45.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:45 smithi072 ceph-mon[41619]: pgmap v1376: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:45 smithi173 ceph-mon[33839]: pgmap v1376: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:47.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:47 smithi072 ceph-mon[30177]: pgmap v1377: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:47.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:47 smithi072 ceph-mon[41619]: pgmap v1377: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:47.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:47 smithi173 ceph-mon[33839]: pgmap v1377: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:49 smithi072 ceph-mon[30177]: pgmap v1378: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:49.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:49 smithi072 ceph-mon[41619]: pgmap v1378: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:49.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:49 smithi173 ceph-mon[33839]: pgmap v1378: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:51 smithi072 ceph-mon[30177]: pgmap v1379: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:24:51.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:51 smithi072 ceph-mon[41619]: pgmap v1379: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:24:51.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:51 smithi173 ceph-mon[33839]: pgmap v1379: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:24:52.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:24:51 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:24:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:53 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:53 smithi072 ceph-mon[30177]: pgmap v1380: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:53 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:24:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:53 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:53 smithi072 ceph-mon[41619]: pgmap v1380: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:53 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:24:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:53 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:24:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:53 smithi173 ceph-mon[33839]: pgmap v1380: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:53 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:24:55.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:55 smithi072 ceph-mon[30177]: pgmap v1381: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:55.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:24:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:24:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:24:55.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:55 smithi072 ceph-mon[41619]: pgmap v1381: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:55.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:55 smithi173 ceph-mon[33839]: pgmap v1381: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:57.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:57 smithi072 ceph-mon[30177]: pgmap v1382: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:57.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:57 smithi072 ceph-mon[41619]: pgmap v1382: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:57.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:57 smithi173 ceph-mon[33839]: pgmap v1382: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:24:59.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:59 smithi072 ceph-mon[30177]: pgmap v1383: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:59.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:24:59 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:24:59.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:59 smithi072 ceph-mon[41619]: pgmap v1383: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:59.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:24:59 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:24:59.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:59 smithi173 ceph-mon[33839]: pgmap v1383: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:24:59.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:24:59 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:25:01.382 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:01 smithi173 ceph-mon[33839]: pgmap v1384: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:01.430 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:01 smithi072 ceph-mon[30177]: pgmap v1384: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:01.430 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:01 smithi072 ceph-mon[41619]: pgmap v1384: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:02.031 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:25:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:25:03.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:02 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:03.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:02 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:25:03.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:02 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:25:03.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:02 smithi173 ceph-mon[33839]: pgmap v1385: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:02 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:02 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:25:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:02 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:25:03.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:02 smithi072 ceph-mon[30177]: pgmap v1385: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:03.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:02 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:03.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:02 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:25:03.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:02 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:25:03.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:02 smithi072 ceph-mon[41619]: pgmap v1385: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:04.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:04 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:25:04.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:04 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:25:04.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:04 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:25:04.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:04 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:25:04.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:04 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:25:04.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:04 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:25:04.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:04 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:25:04.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:04 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:25:04.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:04 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:25:04.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:04 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:25:04.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:04 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:25:04.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:04 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:25:04.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:04 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:25:04.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:04 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:25:04.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:04 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:25:05.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:05 smithi072 ceph-mon[30177]: pgmap v1386: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:25:05.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:25:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:25:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:25:05.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:05 smithi072 ceph-mon[41619]: pgmap v1386: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:25:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:05 smithi173 ceph-mon[33839]: pgmap v1386: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:25:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:07 smithi173 ceph-mon[33839]: pgmap v1387: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:07.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:07 smithi072 ceph-mon[30177]: pgmap v1387: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:07.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:07 smithi072 ceph-mon[41619]: pgmap v1387: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:25:08.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:25:08.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:25:09.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:09 smithi173 ceph-mon[33839]: pgmap v1388: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:09 smithi072 ceph-mon[30177]: pgmap v1388: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:09.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:09 smithi072 ceph-mon[41619]: pgmap v1388: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:11.521 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:11 smithi072 ceph-mon[30177]: pgmap v1389: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:11.521 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:11 smithi072 ceph-mon[41619]: pgmap v1389: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:11.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:11 smithi173 ceph-mon[33839]: pgmap v1389: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:12.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:25:11 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:25:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:13 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:13 smithi173 ceph-mon[33839]: pgmap v1390: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:13.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:13 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:13.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:13 smithi072 ceph-mon[30177]: pgmap v1390: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:13.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:13 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:13.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:13 smithi072 ceph-mon[41619]: pgmap v1390: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:15.239 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:25:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:25:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:25:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:15 smithi173 ceph-mon[33839]: pgmap v1391: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:15.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:15 smithi072 ceph-mon[30177]: pgmap v1391: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:15.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:15 smithi072 ceph-mon[41619]: pgmap v1391: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:17.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:17 smithi173 ceph-mon[33839]: pgmap v1392: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:17.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:17 smithi072 ceph-mon[30177]: pgmap v1392: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:17.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:17 smithi072 ceph-mon[41619]: pgmap v1392: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:19.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:19 smithi173 ceph-mon[33839]: pgmap v1393: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:19.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:19 smithi072 ceph-mon[30177]: pgmap v1393: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:19.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:19 smithi072 ceph-mon[41619]: pgmap v1393: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:21 smithi173 ceph-mon[33839]: pgmap v1394: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:21.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:21 smithi072 ceph-mon[30177]: pgmap v1394: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:21.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:21 smithi072 ceph-mon[41619]: pgmap v1394: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:22.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:25:21 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:25:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:23 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:23 smithi173 ceph-mon[33839]: pgmap v1395: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:23 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:25:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:23 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:23 smithi072 ceph-mon[30177]: pgmap v1395: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:23 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:25:23.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:23 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:23.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:23 smithi072 ceph-mon[41619]: pgmap v1395: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:23.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:23 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:25:25.256 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:25:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:25:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:25:25.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:25 smithi173 ceph-mon[33839]: pgmap v1396: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:25:25.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:25 smithi072 ceph-mon[41619]: pgmap v1396: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:25:25.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:25 smithi072 ceph-mon[30177]: pgmap v1396: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:25:27.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:27 smithi173 ceph-mon[33839]: pgmap v1397: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:27.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:27 smithi072 ceph-mon[41619]: pgmap v1397: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:27.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:27 smithi072 ceph-mon[30177]: pgmap v1397: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:29.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:29 smithi173 ceph-mon[33839]: pgmap v1398: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:25:29.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:29 smithi072 ceph-mon[30177]: pgmap v1398: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:25:29.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:29 smithi072 ceph-mon[41619]: pgmap v1398: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:25:31.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:31 smithi173 ceph-mon[33839]: pgmap v1399: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:31.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:31 smithi072 ceph-mon[30177]: pgmap v1399: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:31.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:31 smithi072 ceph-mon[41619]: pgmap v1399: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:32.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:25:31 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:25:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:33 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:33 smithi173 ceph-mon[33839]: pgmap v1400: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:33.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:33 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:33.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:33 smithi072 ceph-mon[30177]: pgmap v1400: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:33.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:33 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:33.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:33 smithi072 ceph-mon[41619]: pgmap v1400: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:35.270 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:25:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:25:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:25:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:35 smithi173 ceph-mon[33839]: pgmap v1401: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:25:35.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:35 smithi072 ceph-mon[30177]: pgmap v1401: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:25:35.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:35 smithi072 ceph-mon[41619]: pgmap v1401: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:25:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:37 smithi173 ceph-mon[33839]: pgmap v1402: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:37.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:37 smithi072 ceph-mon[41619]: pgmap v1402: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:37.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:37 smithi072 ceph-mon[30177]: pgmap v1402: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:38.565 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:38 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:25:38.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:38 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:25:38.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:38 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:25:39.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:39 smithi173 ceph-mon[33839]: pgmap v1403: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:25:39.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:39 smithi072 ceph-mon[30177]: pgmap v1403: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:25:39.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:39 smithi072 ceph-mon[41619]: pgmap v1403: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:25:41.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:41 smithi173 ceph-mon[33839]: pgmap v1404: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:41.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:41 smithi072 ceph-mon[30177]: pgmap v1404: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:41.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:41 smithi072 ceph-mon[41619]: pgmap v1404: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:42.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:25:41 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:25:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:43 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:43 smithi173 ceph-mon[33839]: pgmap v1405: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:43.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:43 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:43.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:43 smithi072 ceph-mon[30177]: pgmap v1405: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:43.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:43 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:43.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:43 smithi072 ceph-mon[41619]: pgmap v1405: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:45.288 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:25:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:25:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:25:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:45 smithi173 ceph-mon[33839]: pgmap v1406: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:25:45.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:45 smithi072 ceph-mon[30177]: pgmap v1406: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:25:45.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:45 smithi072 ceph-mon[41619]: pgmap v1406: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:25:47.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:47 smithi173 ceph-mon[33839]: pgmap v1407: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:47.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:47 smithi072 ceph-mon[30177]: pgmap v1407: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:47.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:47 smithi072 ceph-mon[41619]: pgmap v1407: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:49.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:49 smithi173 ceph-mon[33839]: pgmap v1408: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:25:49.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:49 smithi072 ceph-mon[30177]: pgmap v1408: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:25:49.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:49 smithi072 ceph-mon[41619]: pgmap v1408: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:25:51.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:51 smithi173 ceph-mon[33839]: pgmap v1409: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:51.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:51 smithi072 ceph-mon[30177]: pgmap v1409: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:51.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:51 smithi072 ceph-mon[41619]: pgmap v1409: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:25:52.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:25:51 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:25:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:53 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:53 smithi173 ceph-mon[33839]: pgmap v1410: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:53 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:25:53.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:53 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:53.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:53 smithi072 ceph-mon[30177]: pgmap v1410: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:53.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:53 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:25:53.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:53 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:25:53.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:53 smithi072 ceph-mon[41619]: pgmap v1410: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:53.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:53 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:25:55.303 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:25:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:25:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:25:55.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:55 smithi173 ceph-mon[33839]: pgmap v1411: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:25:55.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:55 smithi072 ceph-mon[30177]: pgmap v1411: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:25:55.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:55 smithi072 ceph-mon[41619]: pgmap v1411: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:25:57.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:57 smithi173 ceph-mon[33839]: pgmap v1412: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:57.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:57 smithi072 ceph-mon[30177]: pgmap v1412: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:57.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:57 smithi072 ceph-mon[41619]: pgmap v1412: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:25:59.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:25:59 smithi173 ceph-mon[33839]: pgmap v1413: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:25:59.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:25:59 smithi072 ceph-mon[30177]: pgmap v1413: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:25:59.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:25:59 smithi072 ceph-mon[41619]: pgmap v1413: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:01 smithi072 ceph-mon[30177]: pgmap v1414: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:01.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:01 smithi072 ceph-mon[41619]: pgmap v1414: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:01.734 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:01 smithi173 ceph-mon[33839]: pgmap v1414: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:02.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:26:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:26:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:03 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:03 smithi072 ceph-mon[30177]: pgmap v1415: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:03.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:03 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:03.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:03 smithi072 ceph-mon[41619]: pgmap v1415: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:03 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:03 smithi173 ceph-mon[33839]: pgmap v1415: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:04.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:04 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:26:04.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:04 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:26:04.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:04 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:26:05.320 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:26:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:26:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:26:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:05 smithi072 ceph-mon[30177]: pgmap v1416: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:05 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:26:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:05 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:26:05.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:05 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:26:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:05 smithi072 ceph-mon[41619]: pgmap v1416: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:05 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:26:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:05 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:26:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:05 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:26:05.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:05 smithi173 ceph-mon[33839]: pgmap v1416: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:05.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:05 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:26:05.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:05 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:26:05.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:05 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:26:07.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:07 smithi072 ceph-mon[30177]: pgmap v1417: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:07.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:07 smithi072 ceph-mon[41619]: pgmap v1417: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:07 smithi173 ceph-mon[33839]: pgmap v1417: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:08.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:26:08.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:26:08.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:26:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:09 smithi072 ceph-mon[30177]: pgmap v1418: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:09.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:09 smithi072 ceph-mon[41619]: pgmap v1418: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:09.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:09 smithi173 ceph-mon[33839]: pgmap v1418: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:11.615 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:11 smithi173 ceph-mon[33839]: pgmap v1419: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:11.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:11 smithi072 ceph-mon[30177]: pgmap v1419: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:11.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:11 smithi072 ceph-mon[41619]: pgmap v1419: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:12.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:26:11 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:26:13.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:13 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:13.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:13 smithi072 ceph-mon[30177]: pgmap v1420: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:13.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:13 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:13.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:13 smithi072 ceph-mon[41619]: pgmap v1420: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:13.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:13 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:13.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:13 smithi173 ceph-mon[33839]: pgmap v1420: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:15.339 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:26:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:26:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:26:15.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:15 smithi072 ceph-mon[30177]: pgmap v1421: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:15.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:15 smithi072 ceph-mon[41619]: pgmap v1421: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:15.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:15 smithi173 ceph-mon[33839]: pgmap v1421: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:17.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:17 smithi072 ceph-mon[30177]: pgmap v1422: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:17.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:17 smithi072 ceph-mon[41619]: pgmap v1422: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:17.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:17 smithi173 ceph-mon[33839]: pgmap v1422: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:19.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:19 smithi072 ceph-mon[30177]: pgmap v1423: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:19.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:19 smithi072 ceph-mon[41619]: pgmap v1423: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:19 smithi173 ceph-mon[33839]: pgmap v1423: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:21.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:21 smithi072 ceph-mon[30177]: pgmap v1424: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:21.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:21 smithi072 ceph-mon[41619]: pgmap v1424: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:21.756 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:21 smithi173 ceph-mon[33839]: pgmap v1424: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:22.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:26:21 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:26:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:23 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:23 smithi072 ceph-mon[30177]: pgmap v1425: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:23 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:26:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:23 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:23 smithi072 ceph-mon[41619]: pgmap v1425: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:23 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:26:23.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:23 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:23.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:23 smithi173 ceph-mon[33839]: pgmap v1425: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:23.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:23 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:26:25.356 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:26:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:26:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:26:25.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:25 smithi072 ceph-mon[30177]: pgmap v1426: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:25.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:25 smithi072 ceph-mon[41619]: pgmap v1426: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:25.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:25 smithi173 ceph-mon[33839]: pgmap v1426: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:27.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:27 smithi072 ceph-mon[30177]: pgmap v1427: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:27.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:27 smithi072 ceph-mon[41619]: pgmap v1427: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:27.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:27 smithi173 ceph-mon[33839]: pgmap v1427: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:29.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:29 smithi072 ceph-mon[30177]: pgmap v1428: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:29.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:29 smithi072 ceph-mon[41619]: pgmap v1428: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:29.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:29 smithi173 ceph-mon[33839]: pgmap v1428: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:31.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:31 smithi072 ceph-mon[30177]: pgmap v1429: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:31.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:31 smithi072 ceph-mon[41619]: pgmap v1429: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:31.767 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:31 smithi173 ceph-mon[33839]: pgmap v1429: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:32.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:26:31 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:26:33.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:33 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:33.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:33 smithi072 ceph-mon[30177]: pgmap v1430: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:33.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:33 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:33.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:33 smithi072 ceph-mon[41619]: pgmap v1430: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:33.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:33 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:33.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:33 smithi173 ceph-mon[33839]: pgmap v1430: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:35.370 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:26:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:26:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:26:35.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:35 smithi072 ceph-mon[30177]: pgmap v1431: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:35.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:35 smithi072 ceph-mon[41619]: pgmap v1431: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:35.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:35 smithi173 ceph-mon[33839]: pgmap v1431: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:37.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:37 smithi072 ceph-mon[30177]: pgmap v1432: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:37.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:37 smithi072 ceph-mon[41619]: pgmap v1432: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:37.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:37 smithi173 ceph-mon[33839]: pgmap v1432: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:38.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:38 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:26:38.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:38 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:26:38.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:38 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:26:39.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:39 smithi072 ceph-mon[30177]: pgmap v1433: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:39.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:39 smithi072 ceph-mon[41619]: pgmap v1433: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:39.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:39 smithi173 ceph-mon[33839]: pgmap v1433: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:41.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:41 smithi072 ceph-mon[30177]: pgmap v1434: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:41.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:41 smithi072 ceph-mon[41619]: pgmap v1434: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:41.777 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:41 smithi173 ceph-mon[33839]: pgmap v1434: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:26:42.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:26:41 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:26:43.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:43 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:43.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:43 smithi072 ceph-mon[30177]: pgmap v1435: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:43.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:43 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:43.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:43 smithi072 ceph-mon[41619]: pgmap v1435: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:43.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:43 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:43.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:43 smithi173 ceph-mon[33839]: pgmap v1435: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:45.388 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:26:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:26:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:26:45.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:45 smithi072 ceph-mon[41619]: pgmap v1436: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:45.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:45 smithi072 ceph-mon[30177]: pgmap v1436: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:45.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:45 smithi173 ceph-mon[33839]: pgmap v1436: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:47.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:47 smithi072 ceph-mon[30177]: pgmap v1437: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:47.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:47 smithi072 ceph-mon[41619]: pgmap v1437: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:47.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:47 smithi173 ceph-mon[33839]: pgmap v1437: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:49.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:49 smithi072 ceph-mon[30177]: pgmap v1438: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:49.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:49 smithi072 ceph-mon[41619]: pgmap v1438: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:49.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:49 smithi173 ceph-mon[33839]: pgmap v1438: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:51.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:51 smithi072 ceph-mon[30177]: pgmap v1439: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:51.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:51 smithi072 ceph-mon[41619]: pgmap v1439: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:51.779 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:51 smithi173 ceph-mon[33839]: pgmap v1439: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:52.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:26:51 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:26:53.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:53 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:53.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:53 smithi072 ceph-mon[30177]: pgmap v1440: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:53.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:53 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:26:53.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:53 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:53.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:53 smithi072 ceph-mon[41619]: pgmap v1440: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:53.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:53 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:26:53.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:53 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:26:53.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:53 smithi173 ceph-mon[33839]: pgmap v1440: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:53.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:53 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:26:55.405 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:26:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:26:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:26:55.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:55 smithi072 ceph-mon[30177]: pgmap v1441: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:55.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:55 smithi072 ceph-mon[41619]: pgmap v1441: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:55.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:55 smithi173 ceph-mon[33839]: pgmap v1441: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:57.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:57 smithi072 ceph-mon[30177]: pgmap v1442: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:57.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:57 smithi072 ceph-mon[41619]: pgmap v1442: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:57.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:57 smithi173 ceph-mon[33839]: pgmap v1442: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:26:59.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:26:59 smithi072 ceph-mon[30177]: pgmap v1443: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:59.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:26:59 smithi072 ceph-mon[41619]: pgmap v1443: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:26:59.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:26:59 smithi173 ceph-mon[33839]: pgmap v1443: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:01 smithi072 ceph-mon[30177]: pgmap v1444: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:01.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:01 smithi072 ceph-mon[41619]: pgmap v1444: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:01.790 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:01 smithi173 ceph-mon[33839]: pgmap v1444: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:02.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:27:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:27:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:03 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:03 smithi072 ceph-mon[30177]: pgmap v1445: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:03.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:03 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:03.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:03 smithi072 ceph-mon[41619]: pgmap v1445: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:03 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:03 smithi173 ceph-mon[33839]: pgmap v1445: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:05.420 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:27:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:27:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:27:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:05 smithi072 ceph-mon[30177]: pgmap v1446: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:05 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:27:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:05 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:27:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:05 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:27:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:05 smithi072 ceph-mon[41619]: pgmap v1446: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:05 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:27:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:05 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:27:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:05 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:27:05.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:05 smithi173 ceph-mon[33839]: pgmap v1446: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:05.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:05 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:27:05.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:05 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:27:05.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:05 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:27:06.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:06 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:27:06.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:06 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:27:06.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:06 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:27:06.722 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:06 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:27:06.722 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:06 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:27:06.722 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:06 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:27:06.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:06 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:27:06.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:06 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:27:06.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:06 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:27:07.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:07 smithi072 ceph-mon[30177]: pgmap v1447: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:07.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:07 smithi072 ceph-mon[41619]: pgmap v1447: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:07 smithi173 ceph-mon[33839]: pgmap v1447: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:08.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:27:08.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:27:08.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:27:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:09 smithi072 ceph-mon[30177]: pgmap v1448: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:09.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:09 smithi072 ceph-mon[41619]: pgmap v1448: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:09.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:09 smithi173 ceph-mon[33839]: pgmap v1448: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:10.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:10 smithi072 ceph-mon[30177]: pgmap v1449: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:10.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:10 smithi072 ceph-mon[41619]: pgmap v1449: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:10.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:10 smithi173 ceph-mon[33839]: pgmap v1449: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:12.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:27:11 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:27:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:13 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:13 smithi173 ceph-mon[33839]: pgmap v1450: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:13.422 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:13 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:13.422 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:13 smithi072 ceph-mon[30177]: pgmap v1450: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:13.422 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:13 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:13.422 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:13 smithi072 ceph-mon[41619]: pgmap v1450: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:15.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:15 smithi173 ceph-mon[33839]: pgmap v1451: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:15.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:15 smithi072 ceph-mon[30177]: pgmap v1451: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:15.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:15 smithi072 ceph-mon[41619]: pgmap v1451: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:15.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:27:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:27:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:27:17.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:17 smithi173 ceph-mon[33839]: pgmap v1452: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:17.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:17 smithi072 ceph-mon[30177]: pgmap v1452: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:17.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:17 smithi072 ceph-mon[41619]: pgmap v1452: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:19.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:19 smithi173 ceph-mon[33839]: pgmap v1453: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:19.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:19 smithi072 ceph-mon[30177]: pgmap v1453: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:19.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:19 smithi072 ceph-mon[41619]: pgmap v1453: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:21.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:21 smithi173 ceph-mon[33839]: pgmap v1454: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:21.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:21 smithi072 ceph-mon[30177]: pgmap v1454: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:21.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:21 smithi072 ceph-mon[41619]: pgmap v1454: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:22.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:27:21 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:27:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:23 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:23 smithi173 ceph-mon[33839]: pgmap v1455: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:23.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:23 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:27:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:23 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:23 smithi072 ceph-mon[30177]: pgmap v1455: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:23 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:27:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:23 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:23 smithi072 ceph-mon[41619]: pgmap v1455: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:23 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:27:25.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:25 smithi173 ceph-mon[33839]: pgmap v1456: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:25.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:25 smithi072 ceph-mon[30177]: pgmap v1456: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:25.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:27:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:27:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:27:25.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:25 smithi072 ceph-mon[41619]: pgmap v1456: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:27.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:27 smithi173 ceph-mon[33839]: pgmap v1457: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:27 smithi072 ceph-mon[30177]: pgmap v1457: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:27.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:27 smithi072 ceph-mon[41619]: pgmap v1457: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:29.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:29 smithi072 ceph-mon[30177]: pgmap v1458: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:29.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:29 smithi072 ceph-mon[41619]: pgmap v1458: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:29.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:29 smithi173 ceph-mon[33839]: pgmap v1458: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:31.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:31 smithi072 ceph-mon[30177]: pgmap v1459: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:31.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:31 smithi072 ceph-mon[41619]: pgmap v1459: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:31.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:31 smithi173 ceph-mon[33839]: pgmap v1459: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:32.063 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:27:31 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:27:33.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:33 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:33.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:33 smithi072 ceph-mon[30177]: pgmap v1460: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:33.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:33 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:33.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:33 smithi072 ceph-mon[41619]: pgmap v1460: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:33 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:33 smithi173 ceph-mon[33839]: pgmap v1460: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:35.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:35 smithi072 ceph-mon[30177]: pgmap v1461: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:35.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:27:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:27:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:27:35.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:35 smithi072 ceph-mon[41619]: pgmap v1461: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:35 smithi173 ceph-mon[33839]: pgmap v1461: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:36 smithi072 ceph-mon[30177]: pgmap v1462: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:37.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:36 smithi072 ceph-mon[41619]: pgmap v1462: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:36 smithi173 ceph-mon[33839]: pgmap v1462: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:38.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:37 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:27:38.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:37 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:27:38.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:37 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:27:39.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:38 smithi173 ceph-mon[33839]: pgmap v1463: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:39.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:38 smithi072 ceph-mon[30177]: pgmap v1463: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:39.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:38 smithi072 ceph-mon[41619]: pgmap v1463: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:41.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:41 smithi173 ceph-mon[33839]: pgmap v1464: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:41.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:41 smithi072 ceph-mon[30177]: pgmap v1464: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:41.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:41 smithi072 ceph-mon[41619]: pgmap v1464: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:42.215 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:27:41 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:27:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:43 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:43.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:43 smithi173 ceph-mon[33839]: pgmap v1465: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:43 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:43 smithi072 ceph-mon[30177]: pgmap v1465: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:43 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:43 smithi072 ceph-mon[41619]: pgmap v1465: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:45.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:45 smithi173 ceph-mon[33839]: pgmap v1466: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:45.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:45 smithi072 ceph-mon[30177]: pgmap v1466: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:45.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:27:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:27:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:27:45.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:45 smithi072 ceph-mon[41619]: pgmap v1466: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:47.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:47 smithi173 ceph-mon[33839]: pgmap v1467: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:47.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:47 smithi072 ceph-mon[30177]: pgmap v1467: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:47.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:47 smithi072 ceph-mon[41619]: pgmap v1467: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:49 smithi072 ceph-mon[30177]: pgmap v1468: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:49.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:49 smithi072 ceph-mon[41619]: pgmap v1468: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:49.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:49 smithi173 ceph-mon[33839]: pgmap v1468: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:51 smithi072 ceph-mon[30177]: pgmap v1469: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:51.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:51 smithi072 ceph-mon[41619]: pgmap v1469: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:51.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:51 smithi173 ceph-mon[33839]: pgmap v1469: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:27:52.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:27:51 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:27:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:53 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:53 smithi072 ceph-mon[30177]: pgmap v1470: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:53 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:27:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:53 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:53 smithi072 ceph-mon[41619]: pgmap v1470: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:53 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:27:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:53 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:27:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:53 smithi173 ceph-mon[33839]: pgmap v1470: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:53 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:27:55.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:55 smithi072 ceph-mon[30177]: pgmap v1471: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:55.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:27:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:27:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:27:55.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:55 smithi072 ceph-mon[41619]: pgmap v1471: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:55.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:55 smithi173 ceph-mon[33839]: pgmap v1471: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:57.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:57 smithi072 ceph-mon[30177]: pgmap v1472: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:57.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:57 smithi072 ceph-mon[41619]: pgmap v1472: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:57.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:57 smithi173 ceph-mon[33839]: pgmap v1472: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:27:59.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:27:59 smithi072 ceph-mon[30177]: pgmap v1473: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:59.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:27:59 smithi072 ceph-mon[41619]: pgmap v1473: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:27:59.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:27:59 smithi173 ceph-mon[33839]: pgmap v1473: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:01.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:01 smithi072 ceph-mon[30177]: pgmap v1474: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:01.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:01 smithi072 ceph-mon[41619]: pgmap v1474: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:01.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:01 smithi173 ceph-mon[33839]: pgmap v1474: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:02.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:28:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:28:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:03 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:03 smithi072 ceph-mon[30177]: pgmap v1475: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:03.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:03 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:03.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:03 smithi072 ceph-mon[41619]: pgmap v1475: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:03 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:03 smithi173 ceph-mon[33839]: pgmap v1475: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:05.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:05 smithi072 ceph-mon[30177]: pgmap v1476: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:05.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:05 smithi072 ceph-mon[41619]: pgmap v1476: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:05.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:28:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:28:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:28:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:05 smithi173 ceph-mon[33839]: pgmap v1476: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:06.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:06 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:28:06.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:06 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:28:06.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:06 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:28:07.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:07 smithi072 ceph-mon[30177]: pgmap v1477: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:07.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:07 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:28:07.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:07 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:28:07.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:07 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:28:07.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:07 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:28:07.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:07 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:28:07.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:07 smithi072 ceph-mon[41619]: pgmap v1477: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:07.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:07 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:28:07.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:07 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:28:07.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:07 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:28:07.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:07 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:28:07.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:07 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:28:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:07 smithi173 ceph-mon[33839]: pgmap v1477: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:07 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:28:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:07 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:28:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:07 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:28:07.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:07 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:28:07.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:07 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:28:08.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:28:08.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:28:08.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:28:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:09 smithi072 ceph-mon[30177]: pgmap v1478: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:09.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:09 smithi072 ceph-mon[41619]: pgmap v1478: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:09.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:09 smithi173 ceph-mon[33839]: pgmap v1478: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:11.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:11 smithi072 ceph-mon[30177]: pgmap v1479: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:11.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:11 smithi072 ceph-mon[41619]: pgmap v1479: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:11.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:11 smithi173 ceph-mon[33839]: pgmap v1479: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:12.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:28:11 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:28:13.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:13 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:13.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:13 smithi072 ceph-mon[30177]: pgmap v1480: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:13.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:13 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:13.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:13 smithi072 ceph-mon[41619]: pgmap v1480: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:13.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:13 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:13.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:13 smithi173 ceph-mon[33839]: pgmap v1480: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:15.427 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:28:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:28:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:28:15.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:15 smithi072 ceph-mon[30177]: pgmap v1481: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:15.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:15 smithi072 ceph-mon[41619]: pgmap v1481: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:15.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:15 smithi173 ceph-mon[33839]: pgmap v1481: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:17.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:17 smithi072 ceph-mon[30177]: pgmap v1482: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:17.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:17 smithi072 ceph-mon[41619]: pgmap v1482: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:17.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:17 smithi173 ceph-mon[33839]: pgmap v1482: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:19.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:19 smithi072 ceph-mon[30177]: pgmap v1483: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:19.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:19 smithi072 ceph-mon[41619]: pgmap v1483: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:19 smithi173 ceph-mon[33839]: pgmap v1483: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:21.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:21 smithi072 ceph-mon[30177]: pgmap v1484: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:21.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:21 smithi072 ceph-mon[41619]: pgmap v1484: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:21.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:21 smithi173 ceph-mon[33839]: pgmap v1484: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:22.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:28:21 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:28:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:23 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:23 smithi072 ceph-mon[30177]: pgmap v1485: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:23.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:23 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:28:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:23 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:23 smithi072 ceph-mon[41619]: pgmap v1485: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:23 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:28:23.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:23 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:23.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:23 smithi173 ceph-mon[33839]: pgmap v1485: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:23.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:23 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:28:24.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:24 smithi173 ceph-mon[33839]: pgmap v1486: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:24.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:24 smithi072 ceph-mon[30177]: pgmap v1486: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:24.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:24 smithi072 ceph-mon[41619]: pgmap v1486: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:25.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:28:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:28:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:28:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:27 smithi072 ceph-mon[30177]: pgmap v1487: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:27.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:27 smithi072 ceph-mon[41619]: pgmap v1487: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:27.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:27 smithi173 ceph-mon[33839]: pgmap v1487: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:29.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:29 smithi072 ceph-mon[41619]: pgmap v1488: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:29.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:29 smithi072 ceph-mon[30177]: pgmap v1488: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:29.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:29 smithi173 ceph-mon[33839]: pgmap v1488: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:31.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:31 smithi072 ceph-mon[30177]: pgmap v1489: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:31.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:31 smithi072 ceph-mon[41619]: pgmap v1489: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:31.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:31 smithi173 ceph-mon[33839]: pgmap v1489: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:32.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:28:31 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:28:33.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:33 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:33.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:33 smithi072 ceph-mon[30177]: pgmap v1490: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:33.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:33 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:33.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:33 smithi072 ceph-mon[41619]: pgmap v1490: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:33 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:33 smithi173 ceph-mon[33839]: pgmap v1490: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:35.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:35 smithi072 ceph-mon[30177]: pgmap v1491: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:35.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:35 smithi072 ceph-mon[41619]: pgmap v1491: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:35.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:28:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:28:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:28:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:35 smithi173 ceph-mon[33839]: pgmap v1491: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:36 smithi072 ceph-mon[30177]: pgmap v1492: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:37.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:36 smithi072 ceph-mon[41619]: pgmap v1492: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:36 smithi173 ceph-mon[33839]: pgmap v1492: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:38.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:38 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:28:38.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:38 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:28:38.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:38 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:28:39.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:39 smithi072 ceph-mon[30177]: pgmap v1493: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:39.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:39 smithi072 ceph-mon[41619]: pgmap v1493: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:39.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:39 smithi173 ceph-mon[33839]: pgmap v1493: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:41.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:41 smithi072 ceph-mon[30177]: pgmap v1494: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:41.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:41 smithi072 ceph-mon[41619]: pgmap v1494: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:41.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:41 smithi173 ceph-mon[33839]: pgmap v1494: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:42.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:28:41 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:28:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:43 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:43.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:43 smithi072 ceph-mon[30177]: pgmap v1495: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:43 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:43 smithi072 ceph-mon[41619]: pgmap v1495: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:43.525 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:43 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:43.525 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:43 smithi173 ceph-mon[33839]: pgmap v1495: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:45.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:45 smithi072 ceph-mon[30177]: pgmap v1496: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:45.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:28:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:28:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:28:45.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:45 smithi072 ceph-mon[41619]: pgmap v1496: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:45 smithi173 ceph-mon[33839]: pgmap v1496: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:47.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:47 smithi072 ceph-mon[30177]: pgmap v1497: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:47.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:47 smithi072 ceph-mon[41619]: pgmap v1497: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:47.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:47 smithi173 ceph-mon[33839]: pgmap v1497: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:49 smithi072 ceph-mon[30177]: pgmap v1498: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:49.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:49 smithi072 ceph-mon[41619]: pgmap v1498: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:49.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:49 smithi173 ceph-mon[33839]: pgmap v1498: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:51 smithi072 ceph-mon[30177]: pgmap v1499: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:51.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:51 smithi072 ceph-mon[41619]: pgmap v1499: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:51.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:51 smithi173 ceph-mon[33839]: pgmap v1499: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:28:52.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:28:51 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:28:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:53 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:53 smithi173 ceph-mon[33839]: pgmap v1500: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:53 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:28:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:53 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:53 smithi072 ceph-mon[30177]: pgmap v1500: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:53 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:28:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:53 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:28:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:53 smithi072 ceph-mon[41619]: pgmap v1500: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:53 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:28:55.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:55 smithi072 ceph-mon[30177]: pgmap v1501: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:55.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:55 smithi072 ceph-mon[41619]: pgmap v1501: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:55.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:28:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:28:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:28:55.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:55 smithi173 ceph-mon[33839]: pgmap v1501: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:57.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:57 smithi072 ceph-mon[30177]: pgmap v1502: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:57.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:57 smithi072 ceph-mon[41619]: pgmap v1502: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:57.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:57 smithi173 ceph-mon[33839]: pgmap v1502: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:28:59.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:28:59 smithi072 ceph-mon[30177]: pgmap v1503: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:59.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:28:59 smithi072 ceph-mon[41619]: pgmap v1503: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:28:59.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:28:59 smithi173 ceph-mon[33839]: pgmap v1503: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:01.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:01 smithi072 ceph-mon[30177]: pgmap v1504: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:01.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:01 smithi072 ceph-mon[41619]: pgmap v1504: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:01.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:01 smithi173 ceph-mon[33839]: pgmap v1504: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:02.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:29:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:29:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:03 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:03 smithi072 ceph-mon[30177]: pgmap v1505: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:03.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:03 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:03.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:03 smithi072 ceph-mon[41619]: pgmap v1505: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:03 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:03 smithi173 ceph-mon[33839]: pgmap v1505: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:05.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:05 smithi072 ceph-mon[30177]: pgmap v1506: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:05.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:05 smithi072 ceph-mon[41619]: pgmap v1506: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:05.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:29:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:29:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:29:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:05 smithi173 ceph-mon[33839]: pgmap v1506: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:07.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:07 smithi072 ceph-mon[30177]: pgmap v1507: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:07.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:07 smithi072 ceph-mon[41619]: pgmap v1507: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:07 smithi173 ceph-mon[33839]: pgmap v1507: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:08.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:29:08.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:29:08.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:29:08.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:29:08.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:29:08.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:29:08.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:29:08.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:29:08.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:29:08.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:29:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:29:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:29:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:29:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:29:08.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:29:09.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:09 smithi072 ceph-mon[30177]: pgmap v1508: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:09.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:09 smithi072 ceph-mon[41619]: pgmap v1508: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:09.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:09 smithi173 ceph-mon[33839]: pgmap v1508: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:11.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:11 smithi072 ceph-mon[30177]: pgmap v1509: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:11.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:11 smithi072 ceph-mon[41619]: pgmap v1509: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:11.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:11 smithi173 ceph-mon[33839]: pgmap v1509: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:12.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:29:11 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:29:13.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:13 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:13.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:13 smithi072 ceph-mon[30177]: pgmap v1510: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:13.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:13 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:13.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:13 smithi072 ceph-mon[41619]: pgmap v1510: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:13 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:13 smithi173 ceph-mon[33839]: pgmap v1510: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:15.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:15 smithi072 ceph-mon[30177]: pgmap v1511: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:15.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:29:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:29:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:29:15.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:15 smithi072 ceph-mon[41619]: pgmap v1511: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:15 smithi173 ceph-mon[33839]: pgmap v1511: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:17.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:17 smithi072 ceph-mon[30177]: pgmap v1512: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:17.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:17 smithi072 ceph-mon[41619]: pgmap v1512: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:17.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:17 smithi173 ceph-mon[33839]: pgmap v1512: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:19.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:19 smithi072 ceph-mon[30177]: pgmap v1513: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:19.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:19 smithi072 ceph-mon[41619]: pgmap v1513: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:19.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:19 smithi173 ceph-mon[33839]: pgmap v1513: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:21.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:21 smithi072 ceph-mon[30177]: pgmap v1514: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:21.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:21 smithi072 ceph-mon[41619]: pgmap v1514: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:21 smithi173 ceph-mon[33839]: pgmap v1514: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:22.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:29:21 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:29:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:23 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:23 smithi072 ceph-mon[30177]: pgmap v1515: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:23 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:29:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:23 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:23 smithi072 ceph-mon[41619]: pgmap v1515: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:23 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:29:23.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:23 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:23.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:23 smithi173 ceph-mon[33839]: pgmap v1515: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:23.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:23 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:29:25.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:25 smithi072 ceph-mon[30177]: pgmap v1516: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:25.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:25 smithi072 ceph-mon[41619]: pgmap v1516: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:25.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:29:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:29:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:29:25.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:25 smithi173 ceph-mon[33839]: pgmap v1516: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:27 smithi072 ceph-mon[30177]: pgmap v1517: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:27.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:27 smithi072 ceph-mon[41619]: pgmap v1517: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:27.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:27 smithi173 ceph-mon[33839]: pgmap v1517: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:29.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:29 smithi072 ceph-mon[30177]: pgmap v1518: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:29.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:29 smithi072 ceph-mon[41619]: pgmap v1518: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:29.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:29 smithi173 ceph-mon[33839]: pgmap v1518: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:31.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:31 smithi072 ceph-mon[30177]: pgmap v1519: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:31.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:31 smithi072 ceph-mon[41619]: pgmap v1519: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:31.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:31 smithi173 ceph-mon[33839]: pgmap v1519: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:32.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:29:31 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:29:33.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:33 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:33.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:33 smithi072 ceph-mon[30177]: pgmap v1520: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:33.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:33 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:33.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:33 smithi072 ceph-mon[41619]: pgmap v1520: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:33 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:33.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:33 smithi173 ceph-mon[33839]: pgmap v1520: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:35.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:35 smithi072 ceph-mon[30177]: pgmap v1521: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:35.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:29:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:29:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:29:35.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:35 smithi072 ceph-mon[41619]: pgmap v1521: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:35 smithi173 ceph-mon[33839]: pgmap v1521: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:37.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:37 smithi072 ceph-mon[30177]: pgmap v1522: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:37.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:37 smithi072 ceph-mon[41619]: pgmap v1522: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:37 smithi173 ceph-mon[33839]: pgmap v1522: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:38.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:38 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:29:38.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:38 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:29:38.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:38 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:29:39.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:39 smithi072 ceph-mon[30177]: pgmap v1523: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:39.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:39 smithi072 ceph-mon[41619]: pgmap v1523: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:39.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:39 smithi173 ceph-mon[33839]: pgmap v1523: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:41.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:41 smithi072 ceph-mon[30177]: pgmap v1524: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:41.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:41 smithi072 ceph-mon[41619]: pgmap v1524: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:41.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:41 smithi173 ceph-mon[33839]: pgmap v1524: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:42.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:29:41 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:29:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:43 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:43 smithi072 ceph-mon[30177]: pgmap v1525: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:43.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:43 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:43 smithi072 ceph-mon[41619]: pgmap v1525: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:43 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:43 smithi173 ceph-mon[33839]: pgmap v1525: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:45.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:45 smithi072 ceph-mon[30177]: pgmap v1526: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:45.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:45 smithi072 ceph-mon[41619]: pgmap v1526: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:45.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:29:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:29:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:29:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:45 smithi173 ceph-mon[33839]: pgmap v1526: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:47.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:47 smithi072 ceph-mon[30177]: pgmap v1527: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:47.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:47 smithi072 ceph-mon[41619]: pgmap v1527: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:47.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:47 smithi173 ceph-mon[33839]: pgmap v1527: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:49.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:49 smithi072 ceph-mon[41619]: pgmap v1528: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:49 smithi072 ceph-mon[30177]: pgmap v1528: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:49.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:49 smithi173 ceph-mon[33839]: pgmap v1528: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:51 smithi072 ceph-mon[30177]: pgmap v1529: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:51.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:51 smithi072 ceph-mon[41619]: pgmap v1529: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:51.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:51 smithi173 ceph-mon[33839]: pgmap v1529: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:29:52.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:29:51 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:29:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:53 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:53 smithi072 ceph-mon[30177]: pgmap v1530: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:53 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:29:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:53 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:53 smithi072 ceph-mon[41619]: pgmap v1530: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:53 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:29:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:53 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:29:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:53 smithi173 ceph-mon[33839]: pgmap v1530: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:53 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:29:55.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:55 smithi072 ceph-mon[30177]: pgmap v1531: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:55.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:29:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:29:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:29:55.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:55 smithi072 ceph-mon[41619]: pgmap v1531: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:55.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:55 smithi173 ceph-mon[33839]: pgmap v1531: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:57.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:57 smithi072 ceph-mon[30177]: pgmap v1532: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:57.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:57 smithi072 ceph-mon[41619]: pgmap v1532: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:57.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:57 smithi173 ceph-mon[33839]: pgmap v1532: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:29:59.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:29:59 smithi072 ceph-mon[30177]: pgmap v1533: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:59.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:29:59 smithi072 ceph-mon[41619]: pgmap v1533: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:29:59.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:29:59 smithi173 ceph-mon[33839]: pgmap v1533: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:00.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:00 smithi072 ceph-mon[30177]: overall HEALTH_WARN 4 pool(s) do not have an application enabled 2024-03-26T02:30:00.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:00 smithi072 ceph-mon[41619]: overall HEALTH_WARN 4 pool(s) do not have an application enabled 2024-03-26T02:30:00.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:00 smithi173 ceph-mon[33839]: overall HEALTH_WARN 4 pool(s) do not have an application enabled 2024-03-26T02:30:01.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:01 smithi072 ceph-mon[30177]: pgmap v1534: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:01.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:01 smithi072 ceph-mon[41619]: pgmap v1534: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:01.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:01 smithi173 ceph-mon[33839]: pgmap v1534: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:02.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:30:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:30:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:03 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:03 smithi072 ceph-mon[30177]: pgmap v1535: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:03.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:03 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:03.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:03 smithi072 ceph-mon[41619]: pgmap v1535: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:03 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:03 smithi173 ceph-mon[33839]: pgmap v1535: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:05.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:05 smithi072 ceph-mon[30177]: pgmap v1536: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:05.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:05 smithi072 ceph-mon[41619]: pgmap v1536: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:05.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:30:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:30:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:30:05.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:05 smithi173 ceph-mon[33839]: pgmap v1536: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:07 smithi173 ceph-mon[33839]: pgmap v1537: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:07.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:07 smithi072 ceph-mon[30177]: pgmap v1537: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:07.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:07 smithi072 ceph-mon[41619]: pgmap v1537: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:30:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:30:08.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:30:08.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:30:08.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:30:08.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:30:09.308 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:09 smithi072 ceph-mon[30177]: pgmap v1538: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:09.309 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:09 smithi072 ceph-mon[41619]: pgmap v1538: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:09.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:09 smithi173 ceph-mon[33839]: pgmap v1538: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:11.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:11 smithi173 ceph-mon[33839]: pgmap v1539: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:11.602 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:11 smithi072 ceph-mon[30177]: pgmap v1539: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:11.602 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:11 smithi072 ceph-mon[41619]: pgmap v1539: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:12.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:30:11 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:30:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:12 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:12 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:30:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:12 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:30:13.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:12 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:30:13.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:12 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:30:13.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:12 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:30:13.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:12 smithi173 ceph-mon[33839]: pgmap v1540: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:13.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:12 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:13.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:12 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:30:13.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:12 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:30:13.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:12 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:30:13.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:12 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:30:13.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:12 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:30:13.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:12 smithi072 ceph-mon[30177]: pgmap v1540: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:13.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:12 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:13.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:12 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:30:13.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:12 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:30:13.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:12 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:30:13.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:12 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:30:13.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:12 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:30:13.471 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:12 smithi072 ceph-mon[41619]: pgmap v1540: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:15.323 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:15 smithi072 ceph-mon[30177]: pgmap v1541: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:15.323 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:15 smithi072 ceph-mon[41619]: pgmap v1541: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:15.323 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:30:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:30:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:30:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:15 smithi173 ceph-mon[33839]: pgmap v1541: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:17.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:17 smithi072 ceph-mon[30177]: pgmap v1542: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:17.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:17 smithi072 ceph-mon[41619]: pgmap v1542: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:17.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:17 smithi173 ceph-mon[33839]: pgmap v1542: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:19.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:19 smithi072 ceph-mon[30177]: pgmap v1543: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:19.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:19 smithi072 ceph-mon[41619]: pgmap v1543: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:19.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:19 smithi173 ceph-mon[33839]: pgmap v1543: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:21.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:21 smithi072 ceph-mon[30177]: pgmap v1544: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:21.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:21 smithi072 ceph-mon[41619]: pgmap v1544: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:21 smithi173 ceph-mon[33839]: pgmap v1544: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:22.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:30:21 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:30:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:23 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:23 smithi072 ceph-mon[30177]: pgmap v1545: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:23 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:30:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:23 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:23 smithi072 ceph-mon[41619]: pgmap v1545: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:23 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:30:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:23 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:23 smithi173 ceph-mon[33839]: pgmap v1545: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:23 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:30:25.127 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:30:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:30:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:30:25.390 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:25 smithi072 ceph-mon[30177]: pgmap v1546: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:25.390 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:25 smithi072 ceph-mon[41619]: pgmap v1546: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:25.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:25 smithi173 ceph-mon[33839]: pgmap v1546: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:27 smithi072 ceph-mon[30177]: pgmap v1547: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:27.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:27 smithi072 ceph-mon[41619]: pgmap v1547: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:27.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:27 smithi173 ceph-mon[33839]: pgmap v1547: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:29.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:29 smithi072 ceph-mon[30177]: pgmap v1548: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:29.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:29 smithi072 ceph-mon[41619]: pgmap v1548: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:29.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:29 smithi173 ceph-mon[33839]: pgmap v1548: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:31.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:31 smithi072 ceph-mon[30177]: pgmap v1549: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:31.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:31 smithi072 ceph-mon[41619]: pgmap v1549: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:31.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:31 smithi173 ceph-mon[33839]: pgmap v1549: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:32.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:30:31 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:30:33.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:33 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:33.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:33 smithi072 ceph-mon[41619]: pgmap v1550: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:33.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:33 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:33.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:33 smithi072 ceph-mon[30177]: pgmap v1550: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:33 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:33 smithi173 ceph-mon[33839]: pgmap v1550: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:35.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:35 smithi072 ceph-mon[41619]: pgmap v1551: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:35.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:35 smithi072 ceph-mon[30177]: pgmap v1551: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:35.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:30:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:30:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:30:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:35 smithi173 ceph-mon[33839]: pgmap v1551: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:37.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:36 smithi072 ceph-mon[41619]: pgmap v1552: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:36 smithi072 ceph-mon[30177]: pgmap v1552: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:37.315 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:36 smithi173 ceph-mon[33839]: pgmap v1552: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:38.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:38 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:30:38.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:38 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:30:38.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:38 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:30:39.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:39 smithi072 ceph-mon[30177]: pgmap v1553: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:39.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:39 smithi072 ceph-mon[41619]: pgmap v1553: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:39.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:39 smithi173 ceph-mon[33839]: pgmap v1553: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:41.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:41 smithi072 ceph-mon[30177]: pgmap v1554: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:41.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:41 smithi072 ceph-mon[41619]: pgmap v1554: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:41.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:41 smithi173 ceph-mon[33839]: pgmap v1554: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:42.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:30:41 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:30:43.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:43 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:43.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:43 smithi072 ceph-mon[30177]: pgmap v1555: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:43 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:43 smithi072 ceph-mon[41619]: pgmap v1555: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:43 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:43.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:43 smithi173 ceph-mon[33839]: pgmap v1555: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:45.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:45 smithi072 ceph-mon[30177]: pgmap v1556: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:45.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:45 smithi072 ceph-mon[41619]: pgmap v1556: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:45.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:30:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:30:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:30:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:45 smithi173 ceph-mon[33839]: pgmap v1556: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:47.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:47 smithi072 ceph-mon[30177]: pgmap v1557: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:47.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:47 smithi072 ceph-mon[41619]: pgmap v1557: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:47.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:47 smithi173 ceph-mon[33839]: pgmap v1557: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:49 smithi072 ceph-mon[30177]: pgmap v1558: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:49.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:49 smithi072 ceph-mon[41619]: pgmap v1558: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:49.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:49 smithi173 ceph-mon[33839]: pgmap v1558: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:51 smithi072 ceph-mon[30177]: pgmap v1559: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:51.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:51 smithi072 ceph-mon[41619]: pgmap v1559: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:51.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:51 smithi173 ceph-mon[33839]: pgmap v1559: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:30:52.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:30:51 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:30:53.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:53 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:53.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:53 smithi072 ceph-mon[41619]: pgmap v1560: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:53.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:53 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:30:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:53 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:53 smithi072 ceph-mon[30177]: pgmap v1560: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:53 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:30:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:53 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:30:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:53 smithi173 ceph-mon[33839]: pgmap v1560: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:53 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:30:55.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:55 smithi072 ceph-mon[30177]: pgmap v1561: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:55.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:55 smithi072 ceph-mon[41619]: pgmap v1561: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:55.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:30:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:30:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:30:55.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:55 smithi173 ceph-mon[33839]: pgmap v1561: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:57.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:56 smithi072 ceph-mon[30177]: pgmap v1562: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:57.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:56 smithi072 ceph-mon[41619]: pgmap v1562: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:57.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:56 smithi173 ceph-mon[33839]: pgmap v1562: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:30:59.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:30:59 smithi072 ceph-mon[30177]: pgmap v1563: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:59.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:30:59 smithi072 ceph-mon[41619]: pgmap v1563: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:30:59.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:30:59 smithi173 ceph-mon[33839]: pgmap v1563: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:01.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:01 smithi072 ceph-mon[30177]: pgmap v1564: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:01.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:01 smithi072 ceph-mon[41619]: pgmap v1564: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:01.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:01 smithi173 ceph-mon[33839]: pgmap v1564: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:02.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:31:02 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:31:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:03 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:03 smithi072 ceph-mon[30177]: pgmap v1565: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:03.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:03 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:03.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:03 smithi072 ceph-mon[41619]: pgmap v1565: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:03 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:03 smithi173 ceph-mon[33839]: pgmap v1565: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:05.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:05 smithi072 ceph-mon[30177]: pgmap v1566: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:05.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:31:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:31:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:31:05.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:05 smithi072 ceph-mon[41619]: pgmap v1566: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:05 smithi173 ceph-mon[33839]: pgmap v1566: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:07.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:07 smithi072 ceph-mon[30177]: pgmap v1567: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:07.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:07 smithi072 ceph-mon[41619]: pgmap v1567: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:07 smithi173 ceph-mon[33839]: pgmap v1567: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:08.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:31:08.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:31:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:31:09.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:09 smithi072 ceph-mon[30177]: pgmap v1568: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:09.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:09 smithi072 ceph-mon[41619]: pgmap v1568: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:09.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:09 smithi173 ceph-mon[33839]: pgmap v1568: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:11.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:11 smithi072 ceph-mon[30177]: pgmap v1569: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:11.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:11 smithi072 ceph-mon[41619]: pgmap v1569: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:11.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:11 smithi173 ceph-mon[33839]: pgmap v1569: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:12.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:12 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:31:12.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:31:12 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:31:12.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:12 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:31:12.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:12 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:31:13.459 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:13 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:13.459 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:13 smithi173 ceph-mon[33839]: pgmap v1570: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:13.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:13 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:13.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:13 smithi072 ceph-mon[41619]: pgmap v1570: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:13.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:13 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:13.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:13 smithi072 ceph-mon[30177]: pgmap v1570: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:15.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:15 smithi072 ceph-mon[30177]: pgmap v1571: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:15.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:15 smithi072 ceph-mon[41619]: pgmap v1571: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:15.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:31:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:31:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:31:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:15 smithi173 ceph-mon[33839]: pgmap v1571: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:17.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:17 smithi173 ceph-mon[33839]: pgmap v1572: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T02:31:17.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:17 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:31:17.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:17 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:31:17.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:17 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:31:17.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:17 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:31:17.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:17 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:31:17.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:17 smithi072 ceph-mon[30177]: pgmap v1572: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T02:31:17.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:17 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:31:17.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:17 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:31:17.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:17 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:31:17.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:17 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:31:17.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:17 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:31:17.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:17 smithi072 ceph-mon[41619]: pgmap v1572: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T02:31:17.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:17 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:31:17.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:17 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:31:17.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:17 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:31:17.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:17 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:31:17.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:17 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:31:19.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:19 smithi173 ceph-mon[33839]: pgmap v1573: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T02:31:19.723 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:19 smithi072 ceph-mon[30177]: pgmap v1573: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T02:31:19.723 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:19 smithi072 ceph-mon[41619]: pgmap v1573: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T02:31:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:21 smithi173 ceph-mon[33839]: pgmap v1574: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T02:31:21.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:21 smithi072 ceph-mon[30177]: pgmap v1574: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T02:31:21.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:21 smithi072 ceph-mon[41619]: pgmap v1574: 260 pgs: 260 active+clean; 582 KiB data, 766 MiB used, 714 GiB / 715 GiB avail; 4.8 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-26T02:31:22.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:31:22 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:31:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:23 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:23 smithi173 ceph-mon[33839]: pgmap v1575: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T02:31:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:23 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:31:23.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:23 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:23.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:23 smithi072 ceph-mon[41619]: pgmap v1575: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T02:31:23.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:23 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:31:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:23 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:23.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:23 smithi072 ceph-mon[30177]: pgmap v1575: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T02:31:23.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:23 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:31:25.259 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:31:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:31:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:31:25.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:25 smithi173 ceph-mon[33839]: pgmap v1576: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T02:31:25.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:25 smithi072 ceph-mon[30177]: pgmap v1576: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T02:31:25.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:25 smithi072 ceph-mon[41619]: pgmap v1576: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T02:31:27.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:27 smithi173 ceph-mon[33839]: pgmap v1577: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T02:31:27.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:27 smithi072 ceph-mon[30177]: pgmap v1577: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T02:31:27.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:27 smithi072 ceph-mon[41619]: pgmap v1577: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T02:31:29.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:29 smithi173 ceph-mon[33839]: pgmap v1578: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-03-26T02:31:29.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:29 smithi072 ceph-mon[30177]: pgmap v1578: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-03-26T02:31:29.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:29 smithi072 ceph-mon[41619]: pgmap v1578: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-03-26T02:31:31.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:31 smithi173 ceph-mon[33839]: pgmap v1579: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-03-26T02:31:31.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:31 smithi072 ceph-mon[30177]: pgmap v1579: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-03-26T02:31:31.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:31 smithi072 ceph-mon[41619]: pgmap v1579: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 53 op/s 2024-03-26T02:31:32.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:31:32 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:31:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:33 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:33 smithi173 ceph-mon[33839]: pgmap v1580: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-03-26T02:31:33.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:33 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:33.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:33 smithi072 ceph-mon[30177]: pgmap v1580: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-03-26T02:31:33.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:33 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:33.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:33 smithi072 ceph-mon[41619]: pgmap v1580: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 33 KiB/s rd, 0 B/s wr, 54 op/s 2024-03-26T02:31:35.273 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:31:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:31:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:31:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:35 smithi173 ceph-mon[33839]: pgmap v1581: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:35.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:35 smithi072 ceph-mon[30177]: pgmap v1581: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:35.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:35 smithi072 ceph-mon[41619]: pgmap v1581: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:37.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:36 smithi072 ceph-mon[30177]: pgmap v1582: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:37.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:36 smithi072 ceph-mon[41619]: pgmap v1582: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:36 smithi173 ceph-mon[33839]: pgmap v1582: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:38.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:38 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:31:38.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:38 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:31:38.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:38 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:31:39.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:39 smithi072 ceph-mon[30177]: pgmap v1583: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:39.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:39 smithi072 ceph-mon[41619]: pgmap v1583: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:39.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:39 smithi173 ceph-mon[33839]: pgmap v1583: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:41.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:41 smithi072 ceph-mon[30177]: pgmap v1584: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:41.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:41 smithi072 ceph-mon[41619]: pgmap v1584: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:41.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:41 smithi173 ceph-mon[33839]: pgmap v1584: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:42.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:31:42 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:31:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:43 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:43 smithi072 ceph-mon[30177]: pgmap v1585: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:43 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:43.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:43 smithi072 ceph-mon[41619]: pgmap v1585: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:43.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:43 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:43.506 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:43 smithi173 ceph-mon[33839]: pgmap v1585: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:45.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:45 smithi072 ceph-mon[30177]: pgmap v1586: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:45.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:31:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:31:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:31:45.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:45 smithi072 ceph-mon[41619]: pgmap v1586: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:45 smithi173 ceph-mon[33839]: pgmap v1586: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:47.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:47 smithi072 ceph-mon[30177]: pgmap v1587: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:47.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:47 smithi072 ceph-mon[41619]: pgmap v1587: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:47.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:47 smithi173 ceph-mon[33839]: pgmap v1587: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:49 smithi072 ceph-mon[30177]: pgmap v1588: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:49.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:49 smithi072 ceph-mon[41619]: pgmap v1588: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:49.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:49 smithi173 ceph-mon[33839]: pgmap v1588: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:51 smithi072 ceph-mon[30177]: pgmap v1589: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:51.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:51 smithi072 ceph-mon[41619]: pgmap v1589: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:51.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:51 smithi173 ceph-mon[33839]: pgmap v1589: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:31:52.313 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:31:52 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:31:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:53 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:53 smithi173 ceph-mon[33839]: pgmap v1590: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:53.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:53 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:31:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:53 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:53 smithi072 ceph-mon[30177]: pgmap v1590: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:53 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:31:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:53 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:31:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:53 smithi072 ceph-mon[41619]: pgmap v1590: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:53 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:31:55.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:55 smithi072 ceph-mon[30177]: pgmap v1591: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:55.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:55 smithi072 ceph-mon[41619]: pgmap v1591: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:55.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:31:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:31:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:31:55.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:55 smithi173 ceph-mon[33839]: pgmap v1591: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:57.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:57 smithi072 ceph-mon[30177]: pgmap v1592: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:57.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:57 smithi072 ceph-mon[41619]: pgmap v1592: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:57.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:57 smithi173 ceph-mon[33839]: pgmap v1592: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:31:59.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:31:59 smithi072 ceph-mon[30177]: pgmap v1593: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:59.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:31:59 smithi072 ceph-mon[41619]: pgmap v1593: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:31:59.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:31:59 smithi173 ceph-mon[33839]: pgmap v1593: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:01.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:01 smithi072 ceph-mon[30177]: pgmap v1594: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:01.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:01 smithi072 ceph-mon[41619]: pgmap v1594: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:01.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:01 smithi173 ceph-mon[33839]: pgmap v1594: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:02.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:32:02 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:32:03.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:03 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:03.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:03 smithi072 ceph-mon[41619]: pgmap v1595: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:03.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:03 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:03.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:03 smithi072 ceph-mon[30177]: pgmap v1595: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:03 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:03 smithi173 ceph-mon[33839]: pgmap v1595: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:05.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:05 smithi072 ceph-mon[30177]: pgmap v1596: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:05.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:05 smithi072 ceph-mon[41619]: pgmap v1596: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:05.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:32:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:32:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:32:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:05 smithi173 ceph-mon[33839]: pgmap v1596: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:07.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:07 smithi072 ceph-mon[30177]: pgmap v1597: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:07.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:07 smithi072 ceph-mon[41619]: pgmap v1597: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:07 smithi173 ceph-mon[33839]: pgmap v1597: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:32:08.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:32:08.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:32:09.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:09 smithi173 ceph-mon[33839]: pgmap v1598: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:09 smithi072 ceph-mon[30177]: pgmap v1598: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:09.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:09 smithi072 ceph-mon[41619]: pgmap v1598: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:11.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:11 smithi173 ceph-mon[33839]: pgmap v1599: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:11.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:11 smithi072 ceph-mon[30177]: pgmap v1599: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:11.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:11 smithi072 ceph-mon[41619]: pgmap v1599: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:12.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:32:12 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:32:13.521 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:13 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:13.521 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:13 smithi072 ceph-mon[30177]: pgmap v1600: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:13.521 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:13 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:13.521 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:13 smithi072 ceph-mon[41619]: pgmap v1600: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:13 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:13 smithi173 ceph-mon[33839]: pgmap v1600: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:15.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:15 smithi072 ceph-mon[30177]: pgmap v1601: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:15.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:15 smithi072 ceph-mon[41619]: pgmap v1601: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:15.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:32:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:32:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:32:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:15 smithi173 ceph-mon[33839]: pgmap v1601: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:17.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:17 smithi173 ceph-mon[33839]: pgmap v1602: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:17.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:17 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:32:17.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:17 smithi072 ceph-mon[30177]: pgmap v1602: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:17.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:17 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:32:17.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:17 smithi072 ceph-mon[41619]: pgmap v1602: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:17.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:17 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:32:18.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:18 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:32:18.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:18 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:32:18.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:18 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:32:18.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:18 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:32:18.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:18 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:32:18.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:18 smithi173 ceph-mon[33839]: pgmap v1603: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:18.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:18 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:32:18.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:18 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:32:18.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:18 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:32:18.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:18 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:32:18.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:18 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:32:18.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:18 smithi072 ceph-mon[30177]: pgmap v1603: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:18.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:18 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:32:18.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:18 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:32:18.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:18 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:32:18.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:18 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:32:18.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:18 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:32:18.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:18 smithi072 ceph-mon[41619]: pgmap v1603: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:21.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:21 smithi072 ceph-mon[30177]: pgmap v1604: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:21.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:21 smithi072 ceph-mon[41619]: pgmap v1604: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:21 smithi173 ceph-mon[33839]: pgmap v1604: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:22.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:32:22 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:32:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:23 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:23 smithi072 ceph-mon[30177]: pgmap v1605: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:23.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:23 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:32:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:23 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:23 smithi072 ceph-mon[41619]: pgmap v1605: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:23.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:23 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:32:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:23 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:23.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:23 smithi173 ceph-mon[33839]: pgmap v1605: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:23.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:23 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:32:25.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:25 smithi072 ceph-mon[41619]: pgmap v1606: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:25.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:25 smithi072 ceph-mon[30177]: pgmap v1606: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:25.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:32:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:32:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:32:25.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:25 smithi173 ceph-mon[33839]: pgmap v1606: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:27.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:27 smithi072 ceph-mon[30177]: pgmap v1607: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:27.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:27 smithi072 ceph-mon[41619]: pgmap v1607: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:27.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:27 smithi173 ceph-mon[33839]: pgmap v1607: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:29.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:29 smithi072 ceph-mon[30177]: pgmap v1608: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:29.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:29 smithi072 ceph-mon[41619]: pgmap v1608: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:29.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:29 smithi173 ceph-mon[33839]: pgmap v1608: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:31.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:31 smithi072 ceph-mon[30177]: pgmap v1609: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:31.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:31 smithi072 ceph-mon[41619]: pgmap v1609: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:31.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:31 smithi173 ceph-mon[33839]: pgmap v1609: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:32.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:32:32 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:32:33.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:33 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:33.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:33 smithi072 ceph-mon[41619]: pgmap v1610: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:33.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:33 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:33.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:33 smithi072 ceph-mon[30177]: pgmap v1610: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:33 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:33 smithi173 ceph-mon[33839]: pgmap v1610: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:35.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:35 smithi072 ceph-mon[30177]: pgmap v1611: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:35.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:35 smithi072 ceph-mon[41619]: pgmap v1611: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:35.471 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:32:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:32:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:32:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:35 smithi173 ceph-mon[33839]: pgmap v1611: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:37.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:37 smithi072 ceph-mon[30177]: pgmap v1612: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:37.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:37 smithi072 ceph-mon[41619]: pgmap v1612: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:37 smithi173 ceph-mon[33839]: pgmap v1612: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:38.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:38 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:32:38.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:38 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:32:38.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:38 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:32:39.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:39 smithi072 ceph-mon[30177]: pgmap v1613: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:39.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:39 smithi072 ceph-mon[41619]: pgmap v1613: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:39.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:39 smithi173 ceph-mon[33839]: pgmap v1613: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:41.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:41 smithi072 ceph-mon[30177]: pgmap v1614: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:41.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:41 smithi072 ceph-mon[41619]: pgmap v1614: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:41.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:41 smithi173 ceph-mon[33839]: pgmap v1614: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:32:42.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:32:42 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:32:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:43 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:43.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:43 smithi072 ceph-mon[30177]: pgmap v1615: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:43.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:43 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:43.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:43 smithi072 ceph-mon[41619]: pgmap v1615: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:43 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:43 smithi173 ceph-mon[33839]: pgmap v1615: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:45.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:45 smithi072 ceph-mon[30177]: pgmap v1616: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:45.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:45 smithi072 ceph-mon[41619]: pgmap v1616: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:45.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:32:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:32:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:32:45.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:45 smithi173 ceph-mon[33839]: pgmap v1616: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:47.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:47 smithi072 ceph-mon[30177]: pgmap v1617: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:47.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:47 smithi072 ceph-mon[41619]: pgmap v1617: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:47.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:47 smithi173 ceph-mon[33839]: pgmap v1617: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:49.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:49 smithi072 ceph-mon[30177]: pgmap v1618: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:49.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:49 smithi072 ceph-mon[41619]: pgmap v1618: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:49.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:49 smithi173 ceph-mon[33839]: pgmap v1618: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:51.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:51 smithi072 ceph-mon[30177]: pgmap v1619: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:51.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:51 smithi072 ceph-mon[41619]: pgmap v1619: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:51.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:51 smithi173 ceph-mon[33839]: pgmap v1619: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:52.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:32:52 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:32:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:53 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:53 smithi072 ceph-mon[30177]: pgmap v1620: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:53.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:53 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:32:53.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:53 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:53 smithi072 ceph-mon[41619]: pgmap v1620: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:53 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:32:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:53 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:32:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:53 smithi173 ceph-mon[33839]: pgmap v1620: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:53.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:53 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:32:55.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:55 smithi072 ceph-mon[30177]: pgmap v1621: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:55.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:55 smithi072 ceph-mon[41619]: pgmap v1621: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:55.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:32:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:32:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:32:55.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:55 smithi173 ceph-mon[33839]: pgmap v1621: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:57.063 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:32:56 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a[66771]: logger=cleanup t=2024-03-26T02:32:56.640263954Z level=info msg="Completed cleanup jobs" duration=6.891221ms 2024-03-26T02:32:57.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:57 smithi173 ceph-mon[33839]: pgmap v1622: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:57.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:57 smithi072 ceph-mon[30177]: pgmap v1622: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:57.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:57 smithi072 ceph-mon[41619]: pgmap v1622: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:32:59.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:32:59 smithi173 ceph-mon[33839]: pgmap v1623: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:59.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:32:59 smithi072 ceph-mon[30177]: pgmap v1623: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:32:59.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:32:59 smithi072 ceph-mon[41619]: pgmap v1623: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:01.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:01 smithi173 ceph-mon[33839]: pgmap v1624: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:01 smithi072 ceph-mon[30177]: pgmap v1624: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:01.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:01 smithi072 ceph-mon[41619]: pgmap v1624: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:02.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:33:02 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:33:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:03 smithi173 ceph-mon[33839]: pgmap v1625: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:03 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:03 smithi072 ceph-mon[30177]: pgmap v1625: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:03 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:03.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:03 smithi072 ceph-mon[41619]: pgmap v1625: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:03.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:03 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:05.234 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:33:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:33:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:33:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:05 smithi173 ceph-mon[33839]: pgmap v1626: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:33:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:05 smithi072 ceph-mon[30177]: pgmap v1626: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:33:05.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:05 smithi072 ceph-mon[41619]: pgmap v1626: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:33:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:07 smithi173 ceph-mon[33839]: pgmap v1627: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:07.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:07 smithi072 ceph-mon[30177]: pgmap v1627: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:07.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:07 smithi072 ceph-mon[41619]: pgmap v1627: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:33:08.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:33:08.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:33:09.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:09 smithi173 ceph-mon[33839]: pgmap v1628: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:33:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:09 smithi072 ceph-mon[30177]: pgmap v1628: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:33:09.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:09 smithi072 ceph-mon[41619]: pgmap v1628: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:33:11.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:11 smithi173 ceph-mon[33839]: pgmap v1629: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:33:11.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:11 smithi072 ceph-mon[30177]: pgmap v1629: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:33:11.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:11 smithi072 ceph-mon[41619]: pgmap v1629: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T02:33:12.464 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:33:12 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:33:13.521 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:13 smithi072 ceph-mon[30177]: pgmap v1630: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:13.521 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:13 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:13.521 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:13 smithi072 ceph-mon[41619]: pgmap v1630: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:13.521 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:13 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:13 smithi173 ceph-mon[33839]: pgmap v1630: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:13 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:15.252 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:33:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:33:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:33:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:15 smithi173 ceph-mon[33839]: pgmap v1631: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:15.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:15 smithi072 ceph-mon[41619]: pgmap v1631: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:15.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:15 smithi072 ceph-mon[30177]: pgmap v1631: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:17.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:17 smithi173 ceph-mon[33839]: pgmap v1632: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:17.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:17 smithi072 ceph-mon[30177]: pgmap v1632: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:17.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:17 smithi072 ceph-mon[41619]: pgmap v1632: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:18.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:18 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:33:18.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:18 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:33:18.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:18 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:33:19.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:19 smithi173 ceph-mon[33839]: pgmap v1633: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:19.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:19 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:33:19.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:19 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:33:19.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:19 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:33:19.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:19 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:33:19.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:19 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:33:19.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:19 smithi072 ceph-mon[41619]: pgmap v1633: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:19.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:19 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:33:19.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:19 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:33:19.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:19 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:33:19.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:19 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:33:19.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:19 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:33:19.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:19 smithi072 ceph-mon[30177]: pgmap v1633: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:19.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:19 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:33:19.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:19 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:33:19.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:19 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:33:19.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:19 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:33:19.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:19 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:33:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:21 smithi173 ceph-mon[33839]: pgmap v1634: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:21.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:21 smithi072 ceph-mon[30177]: pgmap v1634: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:21.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:21 smithi072 ceph-mon[41619]: pgmap v1634: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:22.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:33:22 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:33:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:23 smithi173 ceph-mon[33839]: pgmap v1635: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:23 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:23 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:33:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:23 smithi072 ceph-mon[30177]: pgmap v1635: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:23 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:23 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:33:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:23 smithi072 ceph-mon[41619]: pgmap v1635: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:23 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:23 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:33:25.272 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:33:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:33:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:33:25.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:25 smithi173 ceph-mon[33839]: pgmap v1636: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:25.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:25 smithi072 ceph-mon[41619]: pgmap v1636: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:25.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:25 smithi072 ceph-mon[30177]: pgmap v1636: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:27.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:27 smithi173 ceph-mon[33839]: pgmap v1637: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:27.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:27 smithi072 ceph-mon[30177]: pgmap v1637: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:27.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:27 smithi072 ceph-mon[41619]: pgmap v1637: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:29.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:29 smithi173 ceph-mon[33839]: pgmap v1638: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:29.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:29 smithi072 ceph-mon[30177]: pgmap v1638: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:29.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:29 smithi072 ceph-mon[41619]: pgmap v1638: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:31.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:31 smithi173 ceph-mon[33839]: pgmap v1639: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:31.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:31 smithi072 ceph-mon[30177]: pgmap v1639: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:31.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:31 smithi072 ceph-mon[41619]: pgmap v1639: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:32.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:33:32 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:33:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:33 smithi173 ceph-mon[33839]: pgmap v1640: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:33 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:33.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:33 smithi072 ceph-mon[30177]: pgmap v1640: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:33.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:33 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:33.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:33 smithi072 ceph-mon[41619]: pgmap v1640: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:33.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:33 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:35.291 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:33:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:33:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:33:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:35 smithi173 ceph-mon[33839]: pgmap v1641: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:35.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:35 smithi072 ceph-mon[30177]: pgmap v1641: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:35.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:35 smithi072 ceph-mon[41619]: pgmap v1641: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:37 smithi173 ceph-mon[33839]: pgmap v1642: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:37.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:37 smithi072 ceph-mon[30177]: pgmap v1642: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:37.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:37 smithi072 ceph-mon[41619]: pgmap v1642: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:38.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:38 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:33:38.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:38 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:33:38.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:38 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:33:39.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:39 smithi173 ceph-mon[33839]: pgmap v1643: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:39.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:39 smithi072 ceph-mon[30177]: pgmap v1643: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:39.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:39 smithi072 ceph-mon[41619]: pgmap v1643: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:41.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:41 smithi173 ceph-mon[33839]: pgmap v1644: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:41.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:41 smithi072 ceph-mon[30177]: pgmap v1644: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:41.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:41 smithi072 ceph-mon[41619]: pgmap v1644: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:42.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:33:42 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:33:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:43 smithi173 ceph-mon[33839]: pgmap v1645: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:43.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:43 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:43.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:43 smithi072 ceph-mon[41619]: pgmap v1645: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:43.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:43 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:43.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:43 smithi072 ceph-mon[30177]: pgmap v1645: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:43.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:43 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:45.312 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:33:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:33:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:33:45.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:45 smithi072 ceph-mon[30177]: pgmap v1646: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:45.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:45 smithi072 ceph-mon[41619]: pgmap v1646: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:45.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:45 smithi173 ceph-mon[33839]: pgmap v1646: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:47.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:47 smithi072 ceph-mon[30177]: pgmap v1647: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:47.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:47 smithi072 ceph-mon[41619]: pgmap v1647: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:47.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:47 smithi173 ceph-mon[33839]: pgmap v1647: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:49.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:49 smithi072 ceph-mon[30177]: pgmap v1648: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:49.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:49 smithi072 ceph-mon[41619]: pgmap v1648: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:49.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:49 smithi173 ceph-mon[33839]: pgmap v1648: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:51.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:51 smithi072 ceph-mon[30177]: pgmap v1649: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:51.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:51 smithi072 ceph-mon[41619]: pgmap v1649: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:51.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:51 smithi173 ceph-mon[33839]: pgmap v1649: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:33:52.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:33:52 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:33:53.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:53 smithi072 ceph-mon[30177]: pgmap v1650: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:53.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:53 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:53.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:53 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:33:53.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:53 smithi072 ceph-mon[41619]: pgmap v1650: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:53.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:53 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:53.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:53 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:33:53.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:53 smithi173 ceph-mon[33839]: pgmap v1650: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:53.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:53 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:33:53.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:53 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:33:55.327 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:33:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:33:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:33:55.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:55 smithi072 ceph-mon[30177]: pgmap v1651: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:55.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:55 smithi072 ceph-mon[41619]: pgmap v1651: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:55.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:55 smithi173 ceph-mon[33839]: pgmap v1651: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:57.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:57 smithi072 ceph-mon[30177]: pgmap v1652: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:57.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:57 smithi072 ceph-mon[41619]: pgmap v1652: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:57.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:57 smithi173 ceph-mon[33839]: pgmap v1652: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:33:58.350 INFO:tasks.workunit.client.0.smithi072.stdout: handler_error: Running main() from gmock_main.cc 2024-03-26T02:33:58.350 INFO:tasks.workunit.client.0.smithi072.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-03-26T02:33:58.350 INFO:tasks.workunit.client.0.smithi072.stdout: handler_error: [----------] Global test environment set-up. 2024-03-26T02:33:58.350 INFO:tasks.workunit.client.0.smithi072.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-03-26T02:33:58.350 INFO:tasks.workunit.client.0.smithi072.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-03-26T02:33:58.350 INFO:tasks.workunit.client.0.smithi072.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (1808987 ms) 2024-03-26T02:33:58.350 INFO:tasks.workunit.client.0.smithi072.stdout: handler_error: [----------] 1 test from neocls_handler_error (1808987 ms total) 2024-03-26T02:33:58.350 INFO:tasks.workunit.client.0.smithi072.stdout: handler_error: 2024-03-26T02:33:58.350 INFO:tasks.workunit.client.0.smithi072.stdout: handler_error: [----------] Global test environment tear-down 2024-03-26T02:33:58.351 INFO:tasks.workunit.client.0.smithi072.stdout: handler_error: [==========] 1 test from 1 test suite ran. (1808987 ms total) 2024-03-26T02:33:58.351 INFO:tasks.workunit.client.0.smithi072.stdout: handler_error: [ PASSED ] 1 test. 2024-03-26T02:33:58.351 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:33:58.351 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83609 2024-03-26T02:33:58.351 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 83609 2024-03-26T02:33:59.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:59 smithi072 ceph-mon[30177]: pgmap v1653: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:59.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:33:59 smithi072 ceph-mon[30177]: osdmap e758: 8 total, 8 up, 8 in 2024-03-26T02:33:59.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:59 smithi072 ceph-mon[41619]: pgmap v1653: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:59.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:33:59 smithi072 ceph-mon[41619]: osdmap e758: 8 total, 8 up, 8 in 2024-03-26T02:33:59.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:59 smithi173 ceph-mon[33839]: pgmap v1653: 260 pgs: 260 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T02:33:59.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:33:59 smithi173 ceph-mon[33839]: osdmap e758: 8 total, 8 up, 8 in 2024-03-26T02:34:00.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:00 smithi072 ceph-mon[30177]: osdmap e759: 8 total, 8 up, 8 in 2024-03-26T02:34:00.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:00 smithi072 ceph-mon[30177]: osdmap e760: 8 total, 8 up, 8 in 2024-03-26T02:34:00.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:00 smithi072 ceph-mon[41619]: osdmap e759: 8 total, 8 up, 8 in 2024-03-26T02:34:00.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:00 smithi072 ceph-mon[41619]: osdmap e760: 8 total, 8 up, 8 in 2024-03-26T02:34:00.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:00 smithi173 ceph-mon[33839]: osdmap e759: 8 total, 8 up, 8 in 2024-03-26T02:34:00.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:00 smithi173 ceph-mon[33839]: osdmap e760: 8 total, 8 up, 8 in 2024-03-26T02:34:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:01 smithi072 ceph-mon[30177]: pgmap v1656: 228 pgs: 32 unknown, 196 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:34:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:01 smithi072 ceph-mon[30177]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:34:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:01 smithi072 ceph-mon[30177]: osdmap e761: 8 total, 8 up, 8 in 2024-03-26T02:34:01.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:01 smithi072 ceph-mon[41619]: pgmap v1656: 228 pgs: 32 unknown, 196 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:34:01.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:01 smithi072 ceph-mon[41619]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:34:01.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:01 smithi072 ceph-mon[41619]: osdmap e761: 8 total, 8 up, 8 in 2024-03-26T02:34:01.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:01 smithi173 ceph-mon[33839]: pgmap v1656: 228 pgs: 32 unknown, 196 active+clean; 582 KiB data, 770 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:34:01.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:01 smithi173 ceph-mon[33839]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:34:01.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:01 smithi173 ceph-mon[33839]: osdmap e761: 8 total, 8 up, 8 in 2024-03-26T02:34:02.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:34:02 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:34:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:03 smithi072 ceph-mon[30177]: pgmap v1659: 228 pgs: 32 unknown, 196 active+clean; 582 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:03 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:03 smithi072 ceph-mon[30177]: osdmap e762: 8 total, 8 up, 8 in 2024-03-26T02:34:03.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:03 smithi072 ceph-mon[41619]: pgmap v1659: 228 pgs: 32 unknown, 196 active+clean; 582 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:03.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:03 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:03.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:03 smithi072 ceph-mon[41619]: osdmap e762: 8 total, 8 up, 8 in 2024-03-26T02:34:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:03 smithi173 ceph-mon[33839]: pgmap v1659: 228 pgs: 32 unknown, 196 active+clean; 582 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:03 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:03 smithi173 ceph-mon[33839]: osdmap e762: 8 total, 8 up, 8 in 2024-03-26T02:34:04.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:04 smithi072 ceph-mon[30177]: osdmap e763: 8 total, 8 up, 8 in 2024-03-26T02:34:04.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:04 smithi072 ceph-mon[41619]: osdmap e763: 8 total, 8 up, 8 in 2024-03-26T02:34:04.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:04 smithi173 ceph-mon[33839]: osdmap e763: 8 total, 8 up, 8 in 2024-03-26T02:34:05.379 INFO:tasks.workunit.client.0.smithi072.stdout: watch_notify: Running main() from gmock_main.cc 2024-03-26T02:34:05.379 INFO:tasks.workunit.client.0.smithi072.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-03-26T02:34:05.379 INFO:tasks.workunit.client.0.smithi072.stdout: watch_notify: [----------] Global test environment set-up. 2024-03-26T02:34:05.379 INFO:tasks.workunit.client.0.smithi072.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-03-26T02:34:05.379 INFO:tasks.workunit.client.0.smithi072.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-03-26T02:34:05.380 INFO:tasks.workunit.client.0.smithi072.stdout: watch_notify: handle_notify cookie 93949391335872 notify_id 3251290243074 notifier_gid 24775 2024-03-26T02:34:05.380 INFO:tasks.workunit.client.0.smithi072.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (1808858 ms) 2024-03-26T02:34:05.380 INFO:tasks.workunit.client.0.smithi072.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-03-26T02:34:05.380 INFO:tasks.workunit.client.0.smithi072.stdout: watch_notify: Trying... 2024-03-26T02:34:05.380 INFO:tasks.workunit.client.0.smithi072.stdout: watch_notify: handle_notify cookie 93949404030912 notify_id 3264175144963 notifier_gid 40979 2024-03-26T02:34:05.380 INFO:tasks.workunit.client.0.smithi072.stdout: watch_notify: Waiting for 3.000000000s 2024-03-26T02:34:05.380 INFO:tasks.workunit.client.0.smithi072.stdout: watch_notify: Timed out. 2024-03-26T02:34:05.380 INFO:tasks.workunit.client.0.smithi072.stdout: watch_notify: Flushing... 2024-03-26T02:34:05.380 INFO:tasks.workunit.client.0.smithi072.stdout: watch_notify: Flushed... 2024-03-26T02:34:05.380 INFO:tasks.workunit.client.0.smithi072.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (7031 ms) 2024-03-26T02:34:05.380 INFO:tasks.workunit.client.0.smithi072.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (1815889 ms total) 2024-03-26T02:34:05.380 INFO:tasks.workunit.client.0.smithi072.stdout: watch_notify: 2024-03-26T02:34:05.380 INFO:tasks.workunit.client.0.smithi072.stdout: watch_notify: [----------] Global test environment tear-down 2024-03-26T02:34:05.380 INFO:tasks.workunit.client.0.smithi072.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (1815889 ms total) 2024-03-26T02:34:05.380 INFO:tasks.workunit.client.0.smithi072.stdout: watch_notify: [ PASSED ] 2 tests. 2024-03-26T02:34:05.381 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:34:05.381 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82624 2024-03-26T02:34:05.381 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 82624 2024-03-26T02:34:05.381 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:34:05.381 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83107 2024-03-26T02:34:05.381 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 83107 2024-03-26T02:34:05.381 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:34:05.381 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83470 2024-03-26T02:34:05.381 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 83470 2024-03-26T02:34:05.381 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:34:05.381 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83185 2024-03-26T02:34:05.381 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 83185 2024-03-26T02:34:05.381 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:34:05.381 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83505 2024-03-26T02:34:05.382 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 83505 2024-03-26T02:34:05.382 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:34:05.382 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83001 2024-03-26T02:34:05.382 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 83001 2024-03-26T02:34:05.382 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:34:05.382 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82489 2024-03-26T02:34:05.382 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 82489 2024-03-26T02:34:05.382 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:34:05.382 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83555 2024-03-26T02:34:05.382 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 83555 2024-03-26T02:34:05.383 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:34:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:34:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:34:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:05 smithi072 ceph-mon[30177]: pgmap v1662: 228 pgs: 32 unknown, 196 active+clean; 582 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:05 smithi072 ceph-mon[30177]: osdmap e764: 8 total, 8 up, 8 in 2024-03-26T02:34:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:05 smithi072 ceph-mon[41619]: pgmap v1662: 228 pgs: 32 unknown, 196 active+clean; 582 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:05.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:05 smithi072 ceph-mon[41619]: osdmap e764: 8 total, 8 up, 8 in 2024-03-26T02:34:05.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:05 smithi173 ceph-mon[33839]: pgmap v1662: 228 pgs: 32 unknown, 196 active+clean; 582 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:05.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:05 smithi173 ceph-mon[33839]: osdmap e764: 8 total, 8 up, 8 in 2024-03-26T02:34:06.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:06 smithi072 ceph-mon[30177]: osdmap e765: 8 total, 8 up, 8 in 2024-03-26T02:34:06.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:06 smithi072 ceph-mon[41619]: osdmap e765: 8 total, 8 up, 8 in 2024-03-26T02:34:06.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:06 smithi173 ceph-mon[33839]: osdmap e765: 8 total, 8 up, 8 in 2024-03-26T02:34:07.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:07 smithi072 ceph-mon[41619]: pgmap v1665: 164 pgs: 164 active+clean; 582 KiB data, 801 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T02:34:07.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:07 smithi072 ceph-mon[41619]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:34:07.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:07 smithi072 ceph-mon[41619]: osdmap e766: 8 total, 8 up, 8 in 2024-03-26T02:34:07.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:07 smithi072 ceph-mon[30177]: pgmap v1665: 164 pgs: 164 active+clean; 582 KiB data, 801 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T02:34:07.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:07 smithi072 ceph-mon[30177]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:34:07.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:07 smithi072 ceph-mon[30177]: osdmap e766: 8 total, 8 up, 8 in 2024-03-26T02:34:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:07 smithi173 ceph-mon[33839]: pgmap v1665: 164 pgs: 164 active+clean; 582 KiB data, 801 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T02:34:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:07 smithi173 ceph-mon[33839]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:34:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:07 smithi173 ceph-mon[33839]: osdmap e766: 8 total, 8 up, 8 in 2024-03-26T02:34:08.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:08 smithi072 ceph-mon[41619]: osdmap e767: 8 total, 8 up, 8 in 2024-03-26T02:34:08.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:34:08.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:08 smithi072 ceph-mon[30177]: osdmap e767: 8 total, 8 up, 8 in 2024-03-26T02:34:08.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:34:08.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:08 smithi173 ceph-mon[33839]: osdmap e767: 8 total, 8 up, 8 in 2024-03-26T02:34:08.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:34:09.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:09 smithi072 ceph-mon[41619]: pgmap v1668: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 801 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T02:34:09.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:09 smithi072 ceph-mon[41619]: osdmap e768: 8 total, 8 up, 8 in 2024-03-26T02:34:09.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:09 smithi072 ceph-mon[30177]: pgmap v1668: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 801 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T02:34:09.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:09 smithi072 ceph-mon[30177]: osdmap e768: 8 total, 8 up, 8 in 2024-03-26T02:34:09.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:09 smithi173 ceph-mon[33839]: pgmap v1668: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 801 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T02:34:09.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:09 smithi173 ceph-mon[33839]: osdmap e768: 8 total, 8 up, 8 in 2024-03-26T02:34:10.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:10 smithi072 ceph-mon[30177]: osdmap e769: 8 total, 8 up, 8 in 2024-03-26T02:34:10.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:10 smithi072 ceph-mon[41619]: osdmap e769: 8 total, 8 up, 8 in 2024-03-26T02:34:10.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:10 smithi173 ceph-mon[33839]: osdmap e769: 8 total, 8 up, 8 in 2024-03-26T02:34:11.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:11 smithi072 ceph-mon[30177]: pgmap v1671: 164 pgs: 164 active+clean; 582 KiB data, 801 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:34:11.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:11 smithi072 ceph-mon[30177]: osdmap e770: 8 total, 8 up, 8 in 2024-03-26T02:34:11.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:11 smithi072 ceph-mon[41619]: pgmap v1671: 164 pgs: 164 active+clean; 582 KiB data, 801 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:34:11.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:11 smithi072 ceph-mon[41619]: osdmap e770: 8 total, 8 up, 8 in 2024-03-26T02:34:11.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:11 smithi173 ceph-mon[33839]: pgmap v1671: 164 pgs: 164 active+clean; 582 KiB data, 801 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:34:11.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:11 smithi173 ceph-mon[33839]: osdmap e770: 8 total, 8 up, 8 in 2024-03-26T02:34:12.445 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:34:12 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:34:12.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:12 smithi072 ceph-mon[30177]: osdmap e771: 8 total, 8 up, 8 in 2024-03-26T02:34:12.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:12 smithi072 ceph-mon[41619]: osdmap e771: 8 total, 8 up, 8 in 2024-03-26T02:34:12.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:12 smithi173 ceph-mon[33839]: osdmap e771: 8 total, 8 up, 8 in 2024-03-26T02:34:13.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:13 smithi072 ceph-mon[41619]: pgmap v1674: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:13.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:13 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:13.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:13 smithi072 ceph-mon[41619]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:34:13.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:13 smithi072 ceph-mon[41619]: osdmap e772: 8 total, 8 up, 8 in 2024-03-26T02:34:13.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:13 smithi072 ceph-mon[30177]: pgmap v1674: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:13.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:13 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:13.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:13 smithi072 ceph-mon[30177]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:34:13.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:13 smithi072 ceph-mon[30177]: osdmap e772: 8 total, 8 up, 8 in 2024-03-26T02:34:13.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:13 smithi173 ceph-mon[33839]: pgmap v1674: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:13.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:13 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:13.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:13 smithi173 ceph-mon[33839]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:34:13.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:13 smithi173 ceph-mon[33839]: osdmap e772: 8 total, 8 up, 8 in 2024-03-26T02:34:14.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:14 smithi072 ceph-mon[30177]: osdmap e773: 8 total, 8 up, 8 in 2024-03-26T02:34:14.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:14 smithi072 ceph-mon[41619]: osdmap e773: 8 total, 8 up, 8 in 2024-03-26T02:34:14.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:14 smithi173 ceph-mon[33839]: osdmap e773: 8 total, 8 up, 8 in 2024-03-26T02:34:15.459 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:34:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:34:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:34:15.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:15 smithi072 ceph-mon[30177]: pgmap v1677: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:15.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:15 smithi072 ceph-mon[30177]: osdmap e774: 8 total, 8 up, 8 in 2024-03-26T02:34:15.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:15 smithi072 ceph-mon[41619]: pgmap v1677: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:15.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:15 smithi072 ceph-mon[41619]: osdmap e774: 8 total, 8 up, 8 in 2024-03-26T02:34:15.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:15 smithi173 ceph-mon[33839]: pgmap v1677: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 849 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:15.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:15 smithi173 ceph-mon[33839]: osdmap e774: 8 total, 8 up, 8 in 2024-03-26T02:34:16.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:16 smithi173 ceph-mon[33839]: osdmap e775: 8 total, 8 up, 8 in 2024-03-26T02:34:16.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:16 smithi072 ceph-mon[30177]: osdmap e775: 8 total, 8 up, 8 in 2024-03-26T02:34:16.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:16 smithi072 ceph-mon[41619]: osdmap e775: 8 total, 8 up, 8 in 2024-03-26T02:34:17.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:17 smithi173 ceph-mon[33839]: pgmap v1680: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 846 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:17.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:17 smithi173 ceph-mon[33839]: osdmap e776: 8 total, 8 up, 8 in 2024-03-26T02:34:17.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:17 smithi072 ceph-mon[30177]: pgmap v1680: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 846 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:17.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:17 smithi072 ceph-mon[30177]: osdmap e776: 8 total, 8 up, 8 in 2024-03-26T02:34:17.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:17 smithi072 ceph-mon[41619]: pgmap v1680: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 846 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:17.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:17 smithi072 ceph-mon[41619]: osdmap e776: 8 total, 8 up, 8 in 2024-03-26T02:34:18.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:18 smithi173 ceph-mon[33839]: osdmap e777: 8 total, 8 up, 8 in 2024-03-26T02:34:18.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:18 smithi072 ceph-mon[30177]: osdmap e777: 8 total, 8 up, 8 in 2024-03-26T02:34:18.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:18 smithi072 ceph-mon[41619]: osdmap e777: 8 total, 8 up, 8 in 2024-03-26T02:34:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:19 smithi173 ceph-mon[33839]: pgmap v1683: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 846 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:19 smithi173 ceph-mon[33839]: osdmap e778: 8 total, 8 up, 8 in 2024-03-26T02:34:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:19 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:34:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:19 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:34:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:19 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:34:19.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:19 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:34:19.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:19 smithi072 ceph-mon[30177]: pgmap v1683: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 846 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:19.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:19 smithi072 ceph-mon[30177]: osdmap e778: 8 total, 8 up, 8 in 2024-03-26T02:34:19.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:19 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:34:19.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:19 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:34:19.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:19 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:34:19.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:19 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:34:19.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:19 smithi072 ceph-mon[41619]: pgmap v1683: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 846 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:19.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:19 smithi072 ceph-mon[41619]: osdmap e778: 8 total, 8 up, 8 in 2024-03-26T02:34:19.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:19 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:34:19.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:19 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:34:19.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:19 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:34:19.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:19 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:34:20.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:20 smithi173 ceph-mon[33839]: osdmap e779: 8 total, 8 up, 8 in 2024-03-26T02:34:20.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:20 smithi173 ceph-mon[33839]: pgmap v1686: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 846 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:34:20.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:20 smithi072 ceph-mon[30177]: osdmap e779: 8 total, 8 up, 8 in 2024-03-26T02:34:20.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:20 smithi072 ceph-mon[30177]: pgmap v1686: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 846 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:34:20.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:20 smithi072 ceph-mon[41619]: osdmap e779: 8 total, 8 up, 8 in 2024-03-26T02:34:20.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:20 smithi072 ceph-mon[41619]: pgmap v1686: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 846 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:34:21.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:21 smithi173 ceph-mon[33839]: osdmap e780: 8 total, 8 up, 8 in 2024-03-26T02:34:21.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:21 smithi072 ceph-mon[30177]: osdmap e780: 8 total, 8 up, 8 in 2024-03-26T02:34:21.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:21 smithi072 ceph-mon[41619]: osdmap e780: 8 total, 8 up, 8 in 2024-03-26T02:34:22.528 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:34:22 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:34:22.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:22 smithi173 ceph-mon[33839]: osdmap e781: 8 total, 8 up, 8 in 2024-03-26T02:34:22.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:22 smithi173 ceph-mon[33839]: pgmap v1689: 164 pgs: 164 active+clean; 582 KiB data, 846 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:22.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:22 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:22.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:22 smithi072 ceph-mon[30177]: osdmap e781: 8 total, 8 up, 8 in 2024-03-26T02:34:22.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:22 smithi072 ceph-mon[30177]: pgmap v1689: 164 pgs: 164 active+clean; 582 KiB data, 846 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:22.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:22 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:22.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:22 smithi072 ceph-mon[41619]: osdmap e781: 8 total, 8 up, 8 in 2024-03-26T02:34:22.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:22 smithi072 ceph-mon[41619]: pgmap v1689: 164 pgs: 164 active+clean; 582 KiB data, 846 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:22.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:22 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:23.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:23 smithi173 ceph-mon[33839]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:34:23.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:23 smithi173 ceph-mon[33839]: osdmap e782: 8 total, 8 up, 8 in 2024-03-26T02:34:23.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:23 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:34:23.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:23 smithi072 ceph-mon[30177]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:34:23.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:23 smithi072 ceph-mon[30177]: osdmap e782: 8 total, 8 up, 8 in 2024-03-26T02:34:23.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:23 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:34:23.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:23 smithi072 ceph-mon[41619]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:34:23.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:23 smithi072 ceph-mon[41619]: osdmap e782: 8 total, 8 up, 8 in 2024-03-26T02:34:23.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:23 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:34:24.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:24 smithi173 ceph-mon[33839]: osdmap e783: 8 total, 8 up, 8 in 2024-03-26T02:34:24.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:24 smithi173 ceph-mon[33839]: pgmap v1692: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 846 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:24.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:24 smithi072 ceph-mon[30177]: osdmap e783: 8 total, 8 up, 8 in 2024-03-26T02:34:24.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:24 smithi072 ceph-mon[30177]: pgmap v1692: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 846 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:24.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:24 smithi072 ceph-mon[41619]: osdmap e783: 8 total, 8 up, 8 in 2024-03-26T02:34:24.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:24 smithi072 ceph-mon[41619]: pgmap v1692: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 846 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:25.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:34:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:34:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:34:25.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:25 smithi173 ceph-mon[33839]: osdmap e784: 8 total, 8 up, 8 in 2024-03-26T02:34:25.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:25 smithi072 ceph-mon[30177]: osdmap e784: 8 total, 8 up, 8 in 2024-03-26T02:34:25.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:25 smithi072 ceph-mon[41619]: osdmap e784: 8 total, 8 up, 8 in 2024-03-26T02:34:26.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:26 smithi173 ceph-mon[33839]: osdmap e785: 8 total, 8 up, 8 in 2024-03-26T02:34:26.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:26 smithi173 ceph-mon[33839]: pgmap v1695: 196 pgs: 196 active+clean; 583 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T02:34:26.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:26 smithi072 ceph-mon[30177]: osdmap e785: 8 total, 8 up, 8 in 2024-03-26T02:34:26.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:26 smithi072 ceph-mon[30177]: pgmap v1695: 196 pgs: 196 active+clean; 583 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T02:34:26.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:26 smithi072 ceph-mon[41619]: osdmap e785: 8 total, 8 up, 8 in 2024-03-26T02:34:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:26 smithi072 ceph-mon[41619]: pgmap v1695: 196 pgs: 196 active+clean; 583 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T02:34:27.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:27 smithi173 ceph-mon[33839]: osdmap e786: 8 total, 8 up, 8 in 2024-03-26T02:34:27.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:27 smithi072 ceph-mon[30177]: osdmap e786: 8 total, 8 up, 8 in 2024-03-26T02:34:27.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:27 smithi072 ceph-mon[41619]: osdmap e786: 8 total, 8 up, 8 in 2024-03-26T02:34:28.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:28 smithi072 ceph-mon[30177]: osdmap e787: 8 total, 8 up, 8 in 2024-03-26T02:34:28.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:28 smithi072 ceph-mon[30177]: pgmap v1698: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:28.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:28 smithi072 ceph-mon[41619]: osdmap e787: 8 total, 8 up, 8 in 2024-03-26T02:34:28.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:28 smithi072 ceph-mon[41619]: pgmap v1698: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:29.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:28 smithi173 ceph-mon[33839]: osdmap e787: 8 total, 8 up, 8 in 2024-03-26T02:34:29.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:28 smithi173 ceph-mon[33839]: pgmap v1698: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 864 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:29.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:29 smithi072 ceph-mon[30177]: osdmap e788: 8 total, 8 up, 8 in 2024-03-26T02:34:29.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:29 smithi072 ceph-mon[41619]: osdmap e788: 8 total, 8 up, 8 in 2024-03-26T02:34:30.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:29 smithi173 ceph-mon[33839]: osdmap e788: 8 total, 8 up, 8 in 2024-03-26T02:34:30.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:30 smithi072 ceph-mon[30177]: osdmap e789: 8 total, 8 up, 8 in 2024-03-26T02:34:30.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:30 smithi072 ceph-mon[30177]: pgmap v1701: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 864 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:34:30.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:30 smithi072 ceph-mon[41619]: osdmap e789: 8 total, 8 up, 8 in 2024-03-26T02:34:30.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:30 smithi072 ceph-mon[41619]: pgmap v1701: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 864 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:34:31.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:30 smithi173 ceph-mon[33839]: osdmap e789: 8 total, 8 up, 8 in 2024-03-26T02:34:31.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:30 smithi173 ceph-mon[33839]: pgmap v1701: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 864 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:34:31.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:31 smithi072 ceph-mon[30177]: osdmap e790: 8 total, 8 up, 8 in 2024-03-26T02:34:31.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:31 smithi072 ceph-mon[30177]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:34:31.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:31 smithi072 ceph-mon[41619]: osdmap e790: 8 total, 8 up, 8 in 2024-03-26T02:34:31.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:31 smithi072 ceph-mon[41619]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:34:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:31 smithi173 ceph-mon[33839]: osdmap e790: 8 total, 8 up, 8 in 2024-03-26T02:34:32.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:31 smithi173 ceph-mon[33839]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:34:32.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:34:32 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:34:32.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:32 smithi072 ceph-mon[30177]: osdmap e791: 8 total, 8 up, 8 in 2024-03-26T02:34:32.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:32 smithi072 ceph-mon[30177]: pgmap v1704: 196 pgs: 196 active+clean; 583 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T02:34:32.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:32 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:32.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:32 smithi072 ceph-mon[41619]: osdmap e791: 8 total, 8 up, 8 in 2024-03-26T02:34:32.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:32 smithi072 ceph-mon[41619]: pgmap v1704: 196 pgs: 196 active+clean; 583 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T02:34:32.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:32 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:33.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:32 smithi173 ceph-mon[33839]: osdmap e791: 8 total, 8 up, 8 in 2024-03-26T02:34:33.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:32 smithi173 ceph-mon[33839]: pgmap v1704: 196 pgs: 196 active+clean; 583 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T02:34:33.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:32 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:33.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:33 smithi072 ceph-mon[30177]: osdmap e792: 8 total, 8 up, 8 in 2024-03-26T02:34:33.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:33 smithi072 ceph-mon[41619]: osdmap e792: 8 total, 8 up, 8 in 2024-03-26T02:34:34.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:33 smithi173 ceph-mon[33839]: osdmap e792: 8 total, 8 up, 8 in 2024-03-26T02:34:34.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:34 smithi072 ceph-mon[30177]: osdmap e793: 8 total, 8 up, 8 in 2024-03-26T02:34:34.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:34 smithi072 ceph-mon[30177]: pgmap v1707: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:34.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:34 smithi072 ceph-mon[41619]: osdmap e793: 8 total, 8 up, 8 in 2024-03-26T02:34:34.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:34 smithi072 ceph-mon[41619]: pgmap v1707: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:35.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:34 smithi173 ceph-mon[33839]: osdmap e793: 8 total, 8 up, 8 in 2024-03-26T02:34:35.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:34 smithi173 ceph-mon[33839]: pgmap v1707: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:35.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:34:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:34:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:34:35.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:35 smithi072 ceph-mon[30177]: osdmap e794: 8 total, 8 up, 8 in 2024-03-26T02:34:35.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:35 smithi072 ceph-mon[41619]: osdmap e794: 8 total, 8 up, 8 in 2024-03-26T02:34:36.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:35 smithi173 ceph-mon[33839]: osdmap e794: 8 total, 8 up, 8 in 2024-03-26T02:34:36.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:36 smithi072 ceph-mon[30177]: osdmap e795: 8 total, 8 up, 8 in 2024-03-26T02:34:36.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:36 smithi072 ceph-mon[30177]: pgmap v1710: 196 pgs: 196 active+clean; 583 KiB data, 882 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:34:36.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:36 smithi072 ceph-mon[41619]: osdmap e795: 8 total, 8 up, 8 in 2024-03-26T02:34:36.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:36 smithi072 ceph-mon[41619]: pgmap v1710: 196 pgs: 196 active+clean; 583 KiB data, 882 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:34:37.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:36 smithi173 ceph-mon[33839]: osdmap e795: 8 total, 8 up, 8 in 2024-03-26T02:34:37.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:36 smithi173 ceph-mon[33839]: pgmap v1710: 196 pgs: 196 active+clean; 583 KiB data, 882 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:34:37.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:37 smithi072 ceph-mon[30177]: osdmap e796: 8 total, 8 up, 8 in 2024-03-26T02:34:37.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:37 smithi072 ceph-mon[41619]: osdmap e796: 8 total, 8 up, 8 in 2024-03-26T02:34:38.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:37 smithi173 ceph-mon[33839]: osdmap e796: 8 total, 8 up, 8 in 2024-03-26T02:34:38.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:38 smithi072 ceph-mon[30177]: osdmap e797: 8 total, 8 up, 8 in 2024-03-26T02:34:38.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:38 smithi072 ceph-mon[30177]: pgmap v1713: 196 pgs: 196 active+clean; 583 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T02:34:38.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:38 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:34:38.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:38 smithi072 ceph-mon[41619]: osdmap e797: 8 total, 8 up, 8 in 2024-03-26T02:34:38.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:38 smithi072 ceph-mon[41619]: pgmap v1713: 196 pgs: 196 active+clean; 583 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T02:34:38.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:38 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:34:39.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:38 smithi173 ceph-mon[33839]: osdmap e797: 8 total, 8 up, 8 in 2024-03-26T02:34:39.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:38 smithi173 ceph-mon[33839]: pgmap v1713: 196 pgs: 196 active+clean; 583 KiB data, 882 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T02:34:39.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:38 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:34:39.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:39 smithi072 ceph-mon[30177]: osdmap e798: 8 total, 8 up, 8 in 2024-03-26T02:34:39.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:39 smithi072 ceph-mon[41619]: osdmap e798: 8 total, 8 up, 8 in 2024-03-26T02:34:40.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:39 smithi173 ceph-mon[33839]: osdmap e798: 8 total, 8 up, 8 in 2024-03-26T02:34:40.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:40 smithi072 ceph-mon[41619]: osdmap e799: 8 total, 8 up, 8 in 2024-03-26T02:34:40.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:40 smithi072 ceph-mon[41619]: pgmap v1716: 196 pgs: 196 active+clean; 583 KiB data, 882 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:34:40.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:40 smithi072 ceph-mon[30177]: osdmap e799: 8 total, 8 up, 8 in 2024-03-26T02:34:40.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:40 smithi072 ceph-mon[30177]: pgmap v1716: 196 pgs: 196 active+clean; 583 KiB data, 882 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:34:41.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:40 smithi173 ceph-mon[33839]: osdmap e799: 8 total, 8 up, 8 in 2024-03-26T02:34:41.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:40 smithi173 ceph-mon[33839]: pgmap v1716: 196 pgs: 196 active+clean; 583 KiB data, 882 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:34:41.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:41 smithi072 ceph-mon[30177]: osdmap e800: 8 total, 8 up, 8 in 2024-03-26T02:34:41.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:41 smithi072 ceph-mon[41619]: osdmap e800: 8 total, 8 up, 8 in 2024-03-26T02:34:42.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:41 smithi173 ceph-mon[33839]: osdmap e800: 8 total, 8 up, 8 in 2024-03-26T02:34:42.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:34:42 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:34:42.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:42 smithi072 ceph-mon[30177]: osdmap e801: 8 total, 8 up, 8 in 2024-03-26T02:34:42.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:42 smithi072 ceph-mon[30177]: pgmap v1719: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 883 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T02:34:42.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:42 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:42.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:42 smithi072 ceph-mon[41619]: osdmap e801: 8 total, 8 up, 8 in 2024-03-26T02:34:42.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:42 smithi072 ceph-mon[41619]: pgmap v1719: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 883 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T02:34:42.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:42 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:43.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:42 smithi173 ceph-mon[33839]: osdmap e801: 8 total, 8 up, 8 in 2024-03-26T02:34:43.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:42 smithi173 ceph-mon[33839]: pgmap v1719: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 883 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T02:34:43.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:42 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:43.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:43 smithi072 ceph-mon[30177]: osdmap e802: 8 total, 8 up, 8 in 2024-03-26T02:34:43.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:43 smithi072 ceph-mon[41619]: osdmap e802: 8 total, 8 up, 8 in 2024-03-26T02:34:44.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:43 smithi173 ceph-mon[33839]: osdmap e802: 8 total, 8 up, 8 in 2024-03-26T02:34:44.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:44 smithi072 ceph-mon[30177]: osdmap e803: 8 total, 8 up, 8 in 2024-03-26T02:34:44.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:44 smithi072 ceph-mon[30177]: pgmap v1722: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 883 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T02:34:44.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:44 smithi072 ceph-mon[41619]: osdmap e803: 8 total, 8 up, 8 in 2024-03-26T02:34:44.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:44 smithi072 ceph-mon[41619]: pgmap v1722: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 883 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T02:34:45.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:44 smithi173 ceph-mon[33839]: osdmap e803: 8 total, 8 up, 8 in 2024-03-26T02:34:45.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:44 smithi173 ceph-mon[33839]: pgmap v1722: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 883 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T02:34:45.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:34:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:34:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:34:45.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:45 smithi072 ceph-mon[30177]: osdmap e804: 8 total, 8 up, 8 in 2024-03-26T02:34:45.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:45 smithi072 ceph-mon[41619]: osdmap e804: 8 total, 8 up, 8 in 2024-03-26T02:34:46.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:45 smithi173 ceph-mon[33839]: osdmap e804: 8 total, 8 up, 8 in 2024-03-26T02:34:46.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:46 smithi072 ceph-mon[30177]: osdmap e805: 8 total, 8 up, 8 in 2024-03-26T02:34:46.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:46 smithi072 ceph-mon[30177]: pgmap v1725: 196 pgs: 196 active+clean; 1.6 MiB data, 886 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 256 KiB/s wr, 1 op/s 2024-03-26T02:34:46.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:46 smithi072 ceph-mon[41619]: osdmap e805: 8 total, 8 up, 8 in 2024-03-26T02:34:46.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:46 smithi072 ceph-mon[41619]: pgmap v1725: 196 pgs: 196 active+clean; 1.6 MiB data, 886 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 256 KiB/s wr, 1 op/s 2024-03-26T02:34:47.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:46 smithi173 ceph-mon[33839]: osdmap e805: 8 total, 8 up, 8 in 2024-03-26T02:34:47.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:46 smithi173 ceph-mon[33839]: pgmap v1725: 196 pgs: 196 active+clean; 1.6 MiB data, 886 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 256 KiB/s wr, 1 op/s 2024-03-26T02:34:47.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:47 smithi072 ceph-mon[30177]: osdmap e806: 8 total, 8 up, 8 in 2024-03-26T02:34:47.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:47 smithi072 ceph-mon[41619]: osdmap e806: 8 total, 8 up, 8 in 2024-03-26T02:34:48.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:47 smithi173 ceph-mon[33839]: osdmap e806: 8 total, 8 up, 8 in 2024-03-26T02:34:48.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:48 smithi072 ceph-mon[30177]: osdmap e807: 8 total, 8 up, 8 in 2024-03-26T02:34:48.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:48 smithi072 ceph-mon[30177]: pgmap v1728: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 886 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:48.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:48 smithi072 ceph-mon[41619]: osdmap e807: 8 total, 8 up, 8 in 2024-03-26T02:34:48.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:48 smithi072 ceph-mon[41619]: pgmap v1728: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 886 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:49.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:48 smithi173 ceph-mon[33839]: osdmap e807: 8 total, 8 up, 8 in 2024-03-26T02:34:49.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:48 smithi173 ceph-mon[33839]: pgmap v1728: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 886 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:50.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:49 smithi173 ceph-mon[33839]: osdmap e808: 8 total, 8 up, 8 in 2024-03-26T02:34:50.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:49 smithi072 ceph-mon[30177]: osdmap e808: 8 total, 8 up, 8 in 2024-03-26T02:34:50.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:49 smithi072 ceph-mon[41619]: osdmap e808: 8 total, 8 up, 8 in 2024-03-26T02:34:51.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:50 smithi173 ceph-mon[33839]: osdmap e809: 8 total, 8 up, 8 in 2024-03-26T02:34:51.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:50 smithi173 ceph-mon[33839]: pgmap v1731: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 886 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:34:51.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:50 smithi072 ceph-mon[30177]: osdmap e809: 8 total, 8 up, 8 in 2024-03-26T02:34:51.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:50 smithi072 ceph-mon[30177]: pgmap v1731: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 886 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:34:51.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:50 smithi072 ceph-mon[41619]: osdmap e809: 8 total, 8 up, 8 in 2024-03-26T02:34:51.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:50 smithi072 ceph-mon[41619]: pgmap v1731: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 886 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:34:52.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:51 smithi173 ceph-mon[33839]: osdmap e810: 8 total, 8 up, 8 in 2024-03-26T02:34:52.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:51 smithi072 ceph-mon[30177]: osdmap e810: 8 total, 8 up, 8 in 2024-03-26T02:34:52.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:51 smithi072 ceph-mon[41619]: osdmap e810: 8 total, 8 up, 8 in 2024-03-26T02:34:52.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:34:52 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:34:53.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:52 smithi173 ceph-mon[33839]: osdmap e811: 8 total, 8 up, 8 in 2024-03-26T02:34:53.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:52 smithi173 ceph-mon[33839]: pgmap v1734: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:53.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:52 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:53.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:52 smithi072 ceph-mon[30177]: osdmap e811: 8 total, 8 up, 8 in 2024-03-26T02:34:53.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:52 smithi072 ceph-mon[30177]: pgmap v1734: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:53.220 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:52 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:52 smithi072 ceph-mon[41619]: osdmap e811: 8 total, 8 up, 8 in 2024-03-26T02:34:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:52 smithi072 ceph-mon[41619]: pgmap v1734: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:53.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:52 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:34:54.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:53 smithi173 ceph-mon[33839]: osdmap e812: 8 total, 8 up, 8 in 2024-03-26T02:34:54.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:53 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:34:54.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:53 smithi072 ceph-mon[30177]: osdmap e812: 8 total, 8 up, 8 in 2024-03-26T02:34:54.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:53 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:34:54.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:53 smithi072 ceph-mon[41619]: osdmap e812: 8 total, 8 up, 8 in 2024-03-26T02:34:54.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:53 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:34:55.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:54 smithi173 ceph-mon[33839]: osdmap e813: 8 total, 8 up, 8 in 2024-03-26T02:34:55.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:54 smithi173 ceph-mon[33839]: pgmap v1737: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:55.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:54 smithi072 ceph-mon[30177]: osdmap e813: 8 total, 8 up, 8 in 2024-03-26T02:34:55.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:54 smithi072 ceph-mon[30177]: pgmap v1737: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:55.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:34:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:34:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:34:55.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:54 smithi072 ceph-mon[41619]: osdmap e813: 8 total, 8 up, 8 in 2024-03-26T02:34:55.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:54 smithi072 ceph-mon[41619]: pgmap v1737: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 884 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:34:56.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:55 smithi173 ceph-mon[33839]: osdmap e814: 8 total, 8 up, 8 in 2024-03-26T02:34:56.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:55 smithi072 ceph-mon[30177]: osdmap e814: 8 total, 8 up, 8 in 2024-03-26T02:34:56.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:55 smithi072 ceph-mon[41619]: osdmap e814: 8 total, 8 up, 8 in 2024-03-26T02:34:57.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:56 smithi173 ceph-mon[33839]: pgmap v1739: 196 pgs: 196 active+clean; 582 KiB data, 888 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 231 B/s wr, 1 op/s 2024-03-26T02:34:57.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:56 smithi072 ceph-mon[30177]: pgmap v1739: 196 pgs: 196 active+clean; 582 KiB data, 888 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 231 B/s wr, 1 op/s 2024-03-26T02:34:57.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:56 smithi072 ceph-mon[41619]: pgmap v1739: 196 pgs: 196 active+clean; 582 KiB data, 888 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 231 B/s wr, 1 op/s 2024-03-26T02:34:59.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:34:59 smithi072 ceph-mon[30177]: pgmap v1740: 196 pgs: 196 active+clean; 582 KiB data, 888 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 0 op/s 2024-03-26T02:34:59.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:34:59 smithi072 ceph-mon[41619]: pgmap v1740: 196 pgs: 196 active+clean; 582 KiB data, 888 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 0 op/s 2024-03-26T02:34:59.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:34:59 smithi173 ceph-mon[33839]: pgmap v1740: 196 pgs: 196 active+clean; 582 KiB data, 888 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 170 B/s wr, 0 op/s 2024-03-26T02:35:01.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:01 smithi173 ceph-mon[33839]: pgmap v1741: 196 pgs: 196 active+clean; 582 KiB data, 888 MiB used, 714 GiB / 715 GiB avail; 690 B/s rd, 138 B/s wr, 0 op/s 2024-03-26T02:35:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:01 smithi072 ceph-mon[30177]: pgmap v1741: 196 pgs: 196 active+clean; 582 KiB data, 888 MiB used, 714 GiB / 715 GiB avail; 690 B/s rd, 138 B/s wr, 0 op/s 2024-03-26T02:35:01.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:01 smithi072 ceph-mon[41619]: pgmap v1741: 196 pgs: 196 active+clean; 582 KiB data, 888 MiB used, 714 GiB / 715 GiB avail; 690 B/s rd, 138 B/s wr, 0 op/s 2024-03-26T02:35:02.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:35:02 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:35:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:03 smithi173 ceph-mon[33839]: pgmap v1742: 196 pgs: 196 active+clean; 582 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 121 B/s wr, 1 op/s 2024-03-26T02:35:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:03 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:03 smithi072 ceph-mon[30177]: pgmap v1742: 196 pgs: 196 active+clean; 582 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 121 B/s wr, 1 op/s 2024-03-26T02:35:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:03 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:03.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:03 smithi072 ceph-mon[41619]: pgmap v1742: 196 pgs: 196 active+clean; 582 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 121 B/s wr, 1 op/s 2024-03-26T02:35:03.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:03 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:05.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:05 smithi072 ceph-mon[30177]: pgmap v1743: 196 pgs: 196 active+clean; 582 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-03-26T02:35:05.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:35:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:35:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:35:05.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:05 smithi072 ceph-mon[41619]: pgmap v1743: 196 pgs: 196 active+clean; 582 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-03-26T02:35:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:05 smithi173 ceph-mon[33839]: pgmap v1743: 196 pgs: 196 active+clean; 582 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-03-26T02:35:07.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:07 smithi173 ceph-mon[33839]: osdmap e815: 8 total, 8 up, 8 in 2024-03-26T02:35:07.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:07 smithi173 ceph-mon[33839]: pgmap v1745: 196 pgs: 196 active+clean; 582 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:35:07.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:07 smithi072 ceph-mon[30177]: osdmap e815: 8 total, 8 up, 8 in 2024-03-26T02:35:07.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:07 smithi072 ceph-mon[30177]: pgmap v1745: 196 pgs: 196 active+clean; 582 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:35:07.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:07 smithi072 ceph-mon[41619]: osdmap e815: 8 total, 8 up, 8 in 2024-03-26T02:35:07.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:07 smithi072 ceph-mon[41619]: pgmap v1745: 196 pgs: 196 active+clean; 582 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:35:09.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:09 smithi173 ceph-mon[33839]: pgmap v1746: 196 pgs: 196 active+clean; 582 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:35:09.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:09 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:35:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:09 smithi072 ceph-mon[30177]: pgmap v1746: 196 pgs: 196 active+clean; 582 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:35:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:09 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:35:09.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:09 smithi072 ceph-mon[41619]: pgmap v1746: 196 pgs: 196 active+clean; 582 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:35:09.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:09 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:35:10.244 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: Running main() from gmock_main.cc 2024-03-26T02:35:10.244 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-03-26T02:35:10.244 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [----------] Global test environment set-up. 2024-03-26T02:35:10.244 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-03-26T02:35:10.244 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-03-26T02:35:10.244 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (1810877 ms) 2024-03-26T02:35:10.244 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-03-26T02:35:10.245 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (5030 ms) 2024-03-26T02:35:10.245 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-03-26T02:35:10.245 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (4037 ms) 2024-03-26T02:35:10.245 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-03-26T02:35:10.245 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (5053 ms) 2024-03-26T02:35:10.245 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-03-26T02:35:10.245 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (7055 ms) 2024-03-26T02:35:10.245 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (1832053 ms total) 2024-03-26T02:35:10.245 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: 2024-03-26T02:35:10.245 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-03-26T02:35:10.245 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-03-26T02:35:10.245 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (5033 ms) 2024-03-26T02:35:10.245 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-03-26T02:35:10.245 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (6045 ms) 2024-03-26T02:35:10.245 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-03-26T02:35:10.246 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (8062 ms) 2024-03-26T02:35:10.246 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-03-26T02:35:10.246 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (6048 ms) 2024-03-26T02:35:10.246 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-03-26T02:35:10.246 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (4031 ms) 2024-03-26T02:35:10.246 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-03-26T02:35:10.246 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi072-83683-11. 2024-03-26T02:35:10.246 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: Waiting for snaps to purge. 2024-03-26T02:35:10.246 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (19496 ms) 2024-03-26T02:35:10.246 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (48715 ms total) 2024-03-26T02:35:10.246 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: 2024-03-26T02:35:10.246 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [----------] Global test environment tear-down 2024-03-26T02:35:10.246 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (1880768 ms total) 2024-03-26T02:35:10.247 INFO:tasks.workunit.client.0.smithi072.stdout: snapshots: [ PASSED ] 11 tests. 2024-03-26T02:35:10.248 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:35:10.248 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82923 2024-03-26T02:35:10.248 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 82923 2024-03-26T02:35:10.248 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:35:10.248 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82318 2024-03-26T02:35:10.248 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 82318 2024-03-26T02:35:10.248 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:35:10.248 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82377 2024-03-26T02:35:10.248 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 82377 2024-03-26T02:35:10.248 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:35:10.248 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82808 2024-03-26T02:35:10.248 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 82808 2024-03-26T02:35:10.248 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:35:10.248 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82417 2024-03-26T02:35:10.248 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 82417 2024-03-26T02:35:10.249 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:35:10.249 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82962 2024-03-26T02:35:10.249 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 82962 2024-03-26T02:35:10.249 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:35:10.249 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=82984 2024-03-26T02:35:10.249 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 82984 2024-03-26T02:35:10.249 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:35:10.249 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83365 2024-03-26T02:35:10.249 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 83365 2024-03-26T02:35:10.249 INFO:tasks.workunit.client.0.smithi072.stderr:+ for t in "${!pids[@]}" 2024-03-26T02:35:10.249 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=83670 2024-03-26T02:35:10.249 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 83670 2024-03-26T02:35:10.249 INFO:tasks.workunit.client.0.smithi072.stderr:+ exit 0 2024-03-26T02:35:10.250 INFO:tasks.workunit.client.0.smithi072.stderr:+ cleanup 2024-03-26T02:35:10.250 INFO:tasks.workunit.client.0.smithi072.stderr:+ pkill -P 82312 2024-03-26T02:35:10.264 INFO:tasks.workunit.client.0.smithi072.stderr:+ true 2024-03-26T02:35:10.264 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-26T02:35:10.264 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-26T02:35:10.296 INFO:tasks.workunit:Running workunits matching rados/test_pool_quota.sh on client.0... 2024-03-26T02:35:10.297 INFO:tasks.workunit:Running workunit rados/test_pool_quota.sh... 2024-03-26T02:35:10.297 DEBUG:teuthology.orchestra.run.smithi072:workunit test rados/test_pool_quota.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9f04c91b77e4619963d41730d8cf756e3a6152de TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_pool_quota.sh 2024-03-26T02:35:10.356 INFO:tasks.workunit.client.0.smithi072.stderr:++ uuidgen 2024-03-26T02:35:10.368 INFO:tasks.workunit.client.0.smithi072.stderr:+ p=de5f38de-babb-481b-9bb5-ced33138378b 2024-03-26T02:35:10.369 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph osd pool create de5f38de-babb-481b-9bb5-ced33138378b 12 2024-03-26T02:35:10.456 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.453+0000 7f9f6b6f4640 1 -- 172.21.15.72:0/3696225106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f64101570 msgr2=0x7f9f641019f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:35:10.456 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.453+0000 7f9f6b6f4640 1 --2- 172.21.15.72:0/3696225106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f64101570 0x7f9f641019f0 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f9f54009860 tx=0x7f9f5402bb70 comp rx=0 tx=0).stop 2024-03-26T02:35:10.456 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.453+0000 7f9f6b6f4640 1 -- 172.21.15.72:0/3696225106 shutdown_connections 2024-03-26T02:35:10.456 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.453+0000 7f9f6b6f4640 1 --2- 172.21.15.72:0/3696225106 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9f64101f30 0x7f9f6410e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:10.457 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.453+0000 7f9f6b6f4640 1 --2- 172.21.15.72:0/3696225106 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f64101570 0x7f9f641019f0 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:10.457 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.453+0000 7f9f6b6f4640 1 --2- 172.21.15.72:0/3696225106 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f64107630 0x7f9f64107a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:10.457 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.453+0000 7f9f6b6f4640 1 -- 172.21.15.72:0/3696225106 >> 172.21.15.72:0/3696225106 conn(0x7f9f640fd020 msgr2=0x7f9f640ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:35:10.457 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.453+0000 7f9f6b6f4640 1 -- 172.21.15.72:0/3696225106 shutdown_connections 2024-03-26T02:35:10.457 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.453+0000 7f9f6b6f4640 1 -- 172.21.15.72:0/3696225106 wait complete. 2024-03-26T02:35:10.457 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.454+0000 7f9f6b6f4640 1 Processor -- start 2024-03-26T02:35:10.457 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.454+0000 7f9f6b6f4640 1 -- start start 2024-03-26T02:35:10.457 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.454+0000 7f9f6b6f4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f64101570 0x7f9f641972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:35:10.457 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.454+0000 7f9f6b6f4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f64101f30 0x7f9f64197830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:35:10.457 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.454+0000 7f9f6b6f4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9f64107630 0x7f9f6419bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:35:10.457 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.454+0000 7f9f6b6f4640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6419c1b0 con 0x7f9f64101570 2024-03-26T02:35:10.457 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.454+0000 7f9f6b6f4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6419c320 con 0x7f9f64101f30 2024-03-26T02:35:10.457 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.454+0000 7f9f6b6f4640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6419c4e0 con 0x7f9f64107630 2024-03-26T02:35:10.458 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.454+0000 7f9f69490640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f64101570 0x7f9f641972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:35:10.458 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.454+0000 7f9f69490640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f64101570 0x7f9f641972f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:52074/0 (socket says 172.21.15.72:52074) 2024-03-26T02:35:10.458 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.454+0000 7f9f69490640 1 -- 172.21.15.72:0/254221309 learned_addr learned my addr 172.21.15.72:0/254221309 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:35:10.458 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.454+0000 7f9f69c91640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9f64107630 0x7f9f6419bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:35:10.458 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.454+0000 7f9f69c91640 1 -- 172.21.15.72:0/254221309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f64101f30 msgr2=0x7f9f64197830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:35:10.458 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.454+0000 7f9f69c91640 1 --2- 172.21.15.72:0/254221309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f64101f30 0x7f9f64197830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:10.458 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.454+0000 7f9f69c91640 1 -- 172.21.15.72:0/254221309 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f64101570 msgr2=0x7f9f641972f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:35:10.458 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.454+0000 7f9f69c91640 1 --2- 172.21.15.72:0/254221309 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f64101570 0x7f9f641972f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:10.458 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.454+0000 7f9f69c91640 1 -- 172.21.15.72:0/254221309 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f5c009900 con 0x7f9f64107630 2024-03-26T02:35:10.458 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.454+0000 7f9f69490640 1 --2- 172.21.15.72:0/254221309 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f64101570 0x7f9f641972f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:35:10.458 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.455+0000 7f9f69c91640 1 --2- 172.21.15.72:0/254221309 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9f64107630 0x7f9f6419bbb0 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c00efc0 tx=0x7f9f5c00c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:35:10.458 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.455+0000 7f9f5a7fc640 1 -- 172.21.15.72:0/254221309 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c003970 con 0x7f9f64107630 2024-03-26T02:35:10.458 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.455+0000 7f9f6b6f4640 1 -- 172.21.15.72:0/254221309 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f54009490 con 0x7f9f64107630 2024-03-26T02:35:10.458 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.455+0000 7f9f5a7fc640 1 -- 172.21.15.72:0/254221309 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f9f5c004020 con 0x7f9f64107630 2024-03-26T02:35:10.459 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.455+0000 7f9f5a7fc640 1 -- 172.21.15.72:0/254221309 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f5c00b850 con 0x7f9f64107630 2024-03-26T02:35:10.459 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.455+0000 7f9f6b6f4640 1 -- 172.21.15.72:0/254221309 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f641a3e60 con 0x7f9f64107630 2024-03-26T02:35:10.459 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.456+0000 7f9f6b6f4640 1 -- 172.21.15.72:0/254221309 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f640688c0 con 0x7f9f64107630 2024-03-26T02:35:10.459 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.456+0000 7f9f5a7fc640 1 -- 172.21.15.72:0/254221309 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7f9f5c00a040 con 0x7f9f64107630 2024-03-26T02:35:10.460 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.457+0000 7f9f5a7fc640 1 --2- 172.21.15.72:0/254221309 >> v2:172.21.15.72:6800/3246690770 conn(0x7f9f5006d9f0 0x7f9f5006feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:35:10.460 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.457+0000 7f9f5a7fc640 1 -- 172.21.15.72:0/254221309 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(816..816 src has 255..816) v4 ==== 7458+0+0 (secure 0 0 0) 0x7f9f5c08db90 con 0x7f9f64107630 2024-03-26T02:35:10.460 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.457+0000 7f9f69490640 1 --2- 172.21.15.72:0/254221309 >> v2:172.21.15.72:6800/3246690770 conn(0x7f9f5006d9f0 0x7f9f5006feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:35:10.460 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.457+0000 7f9f69490640 1 --2- 172.21.15.72:0/254221309 >> v2:172.21.15.72:6800/3246690770 conn(0x7f9f5006d9f0 0x7f9f5006feb0 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 crypto rx=0x7f9f60009e10 tx=0x7f9f600091c0 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:35:10.462 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.462+0000 7f9f5a7fc640 1 -- 172.21.15.72:0/254221309 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f9f5c05ab70 con 0x7f9f64107630 2024-03-26T02:35:10.643 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:10.642+0000 7f9f6b6f4640 1 -- 172.21.15.72:0/254221309 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pg_num": 12} v 0) v1 -- 0x7f9f64110db0 con 0x7f9f64107630 2024-03-26T02:35:11.241 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.240+0000 7f9f5a7fc640 1 -- 172.21.15.72:0/254221309 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pg_num": 12}]=0 pool 'de5f38de-babb-481b-9bb5-ced33138378b' created v817) v1 ==== 176+0+0 (secure 0 0 0) 0x7f9f5c05a700 con 0x7f9f64107630 2024-03-26T02:35:11.347 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.346+0000 7f9f6b6f4640 1 -- 172.21.15.72:0/254221309 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pg_num": 12} v 0) v1 -- 0x7f9f641110a0 con 0x7f9f64107630 2024-03-26T02:35:11.348 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.347+0000 7f9f5a7fc640 1 -- 172.21.15.72:0/254221309 <== mon.2 v2:172.21.15.72:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pg_num": 12}]=0 pool 'de5f38de-babb-481b-9bb5-ced33138378b' already exists v817) v1 ==== 183+0+0 (secure 0 0 0) 0x7f9f5c05a8e0 con 0x7f9f64107630 2024-03-26T02:35:11.348 INFO:tasks.workunit.client.0.smithi072.stderr:pool 'de5f38de-babb-481b-9bb5-ced33138378b' already exists 2024-03-26T02:35:11.351 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.349+0000 7f9f6b6f4640 1 -- 172.21.15.72:0/254221309 >> v2:172.21.15.72:6800/3246690770 conn(0x7f9f5006d9f0 msgr2=0x7f9f5006feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:35:11.351 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.349+0000 7f9f6b6f4640 1 --2- 172.21.15.72:0/254221309 >> v2:172.21.15.72:6800/3246690770 conn(0x7f9f5006d9f0 0x7f9f5006feb0 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 crypto rx=0x7f9f60009e10 tx=0x7f9f600091c0 comp rx=0 tx=0).stop 2024-03-26T02:35:11.351 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.349+0000 7f9f6b6f4640 1 -- 172.21.15.72:0/254221309 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9f64107630 msgr2=0x7f9f6419bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:35:11.351 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.349+0000 7f9f6b6f4640 1 --2- 172.21.15.72:0/254221309 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9f64107630 0x7f9f6419bbb0 secure :-1 s=READY pgs=2530 cs=0 l=1 rev1=1 crypto rx=0x7f9f5c00efc0 tx=0x7f9f5c00c800 comp rx=0 tx=0).stop 2024-03-26T02:35:11.352 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.349+0000 7f9f6b6f4640 1 -- 172.21.15.72:0/254221309 shutdown_connections 2024-03-26T02:35:11.352 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.349+0000 7f9f6b6f4640 1 --2- 172.21.15.72:0/254221309 >> v2:172.21.15.72:6800/3246690770 conn(0x7f9f5006d9f0 0x7f9f5006feb0 unknown :-1 s=CLOSED pgs=3368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:11.352 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.349+0000 7f9f6b6f4640 1 --2- 172.21.15.72:0/254221309 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f9f64107630 0x7f9f6419bbb0 unknown :-1 s=CLOSED pgs=2530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:11.352 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.349+0000 7f9f6b6f4640 1 --2- 172.21.15.72:0/254221309 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9f64101f30 0x7f9f64197830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:11.352 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.349+0000 7f9f6b6f4640 1 --2- 172.21.15.72:0/254221309 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f9f64101570 0x7f9f641972f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:11.353 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.349+0000 7f9f6b6f4640 1 -- 172.21.15.72:0/254221309 >> 172.21.15.72:0/254221309 conn(0x7f9f640fd020 msgr2=0x7f9f64105120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:35:11.353 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.349+0000 7f9f6b6f4640 1 -- 172.21.15.72:0/254221309 shutdown_connections 2024-03-26T02:35:11.353 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.349+0000 7f9f6b6f4640 1 -- 172.21.15.72:0/254221309 wait complete. 2024-03-26T02:35:11.360 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph osd pool set-quota de5f38de-babb-481b-9bb5-ced33138378b max_objects 10 2024-03-26T02:35:11.444 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.441+0000 7f3efb11d640 1 -- 172.21.15.72:0/144701051 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3ef4106550 msgr2=0x7f3ef4106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:35:11.444 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.441+0000 7f3efb11d640 1 --2- 172.21.15.72:0/144701051 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3ef4106550 0x7f3ef4106930 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7f3ee8009890 tx=0x7f3ee802bb70 comp rx=0 tx=0).stop 2024-03-26T02:35:11.444 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.441+0000 7f3efb11d640 1 -- 172.21.15.72:0/144701051 shutdown_connections 2024-03-26T02:35:11.445 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.441+0000 7f3efb11d640 1 --2- 172.21.15.72:0/144701051 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ef4100e50 0x7f3ef410d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:11.445 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.441+0000 7f3efb11d640 1 --2- 172.21.15.72:0/144701051 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ef4100490 0x7f3ef4100910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:11.445 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.441+0000 7f3efb11d640 1 --2- 172.21.15.72:0/144701051 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3ef4106550 0x7f3ef4106930 unknown :-1 s=CLOSED pgs=2531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:11.445 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.441+0000 7f3efb11d640 1 -- 172.21.15.72:0/144701051 >> 172.21.15.72:0/144701051 conn(0x7f3ef4077580 msgr2=0x7f3ef4077990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:35:11.445 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.441+0000 7f3efb11d640 1 -- 172.21.15.72:0/144701051 shutdown_connections 2024-03-26T02:35:11.445 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.441+0000 7f3efb11d640 1 -- 172.21.15.72:0/144701051 wait complete. 2024-03-26T02:35:11.445 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.442+0000 7f3efb11d640 1 Processor -- start 2024-03-26T02:35:11.445 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.442+0000 7f3efb11d640 1 -- start start 2024-03-26T02:35:11.445 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.442+0000 7f3efb11d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ef4100490 0x7f3ef4197290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:35:11.445 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.442+0000 7f3efb11d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3ef4100e50 0x7f3ef41977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:35:11.445 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.442+0000 7f3efb11d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ef4106550 0x7f3ef419bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:35:11.445 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.442+0000 7f3efb11d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef419c100 con 0x7f3ef4106550 2024-03-26T02:35:11.446 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.442+0000 7f3efb11d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef419c270 con 0x7f3ef4100490 2024-03-26T02:35:11.446 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.442+0000 7f3efb11d640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ef419c430 con 0x7f3ef4100e50 2024-03-26T02:35:11.446 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.442+0000 7f3ef3fff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3ef4100e50 0x7f3ef41977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:35:11.446 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.442+0000 7f3ef3fff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3ef4100e50 0x7f3ef41977d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:44800/0 (socket says 172.21.15.72:44800) 2024-03-26T02:35:11.446 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.442+0000 7f3ef96ba640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ef4106550 0x7f3ef419bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:35:11.446 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.442+0000 7f3ef3fff640 1 -- 172.21.15.72:0/3140409301 learned_addr learned my addr 172.21.15.72:0/3140409301 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:35:11.446 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.442+0000 7f3ef8eb9640 1 --2- 172.21.15.72:0/3140409301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ef4100490 0x7f3ef4197290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:35:11.446 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.442+0000 7f3ef3fff640 1 -- 172.21.15.72:0/3140409301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ef4100490 msgr2=0x7f3ef4197290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:35:11.446 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.442+0000 7f3ef3fff640 1 --2- 172.21.15.72:0/3140409301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ef4100490 0x7f3ef4197290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:11.446 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.442+0000 7f3ef3fff640 1 -- 172.21.15.72:0/3140409301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ef4106550 msgr2=0x7f3ef419bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:35:11.446 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.442+0000 7f3ef3fff640 1 --2- 172.21.15.72:0/3140409301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ef4106550 0x7f3ef419bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:11.446 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.442+0000 7f3ef3fff640 1 -- 172.21.15.72:0/3140409301 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eec009900 con 0x7f3ef4100e50 2024-03-26T02:35:11.446 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.442+0000 7f3ef8eb9640 1 --2- 172.21.15.72:0/3140409301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ef4100490 0x7f3ef4197290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T02:35:11.446 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.442+0000 7f3ef96ba640 1 --2- 172.21.15.72:0/3140409301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ef4106550 0x7f3ef419bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:35:11.447 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.442+0000 7f3ef3fff640 1 --2- 172.21.15.72:0/3140409301 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3ef4100e50 0x7f3ef41977d0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7f3ee000c920 tx=0x7f3ee000cde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:35:11.447 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.443+0000 7f3ef1ffb640 1 -- 172.21.15.72:0/3140409301 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0007c70 con 0x7f3ef4100e50 2024-03-26T02:35:11.447 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.443+0000 7f3ef1ffb640 1 -- 172.21.15.72:0/3140409301 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f3ee0007dd0 con 0x7f3ef4100e50 2024-03-26T02:35:11.447 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.443+0000 7f3efb11d640 1 -- 172.21.15.72:0/3140409301 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee8009490 con 0x7f3ef4100e50 2024-03-26T02:35:11.447 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.443+0000 7f3ef1ffb640 1 -- 172.21.15.72:0/3140409301 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee0018720 con 0x7f3ef4100e50 2024-03-26T02:35:11.447 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.443+0000 7f3efb11d640 1 -- 172.21.15.72:0/3140409301 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ef41a3db0 con 0x7f3ef4100e50 2024-03-26T02:35:11.447 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.444+0000 7f3efb11d640 1 -- 172.21.15.72:0/3140409301 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ef40688c0 con 0x7f3ef4100e50 2024-03-26T02:35:11.448 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.444+0000 7f3ef1ffb640 1 -- 172.21.15.72:0/3140409301 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7f3ee000f450 con 0x7f3ef4100e50 2024-03-26T02:35:11.448 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.444+0000 7f3ef1ffb640 1 --2- 172.21.15.72:0/3140409301 >> v2:172.21.15.72:6800/3246690770 conn(0x7f3ee406d9f0 0x7f3ee406feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:35:11.448 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.445+0000 7f3ef1ffb640 1 -- 172.21.15.72:0/3140409301 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(817..817 src has 255..817) v4 ==== 7833+0+0 (secure 0 0 0) 0x7f3ee008db50 con 0x7f3ef4100e50 2024-03-26T02:35:11.448 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.445+0000 7f3ef8eb9640 1 --2- 172.21.15.72:0/3140409301 >> v2:172.21.15.72:6800/3246690770 conn(0x7f3ee406d9f0 0x7f3ee406feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:35:11.448 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.445+0000 7f3ef8eb9640 1 --2- 172.21.15.72:0/3140409301 >> v2:172.21.15.72:6800/3246690770 conn(0x7f3ee406d9f0 0x7f3ee406feb0 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 crypto rx=0x7f3ee80323d0 tx=0x7f3ee8035f30 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:35:11.450 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.449+0000 7f3ef1ffb640 1 -- 172.21.15.72:0/3140409301 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f3ee005a9d0 con 0x7f3ef4100e50 2024-03-26T02:35:11.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:11 smithi173 ceph-mon[33839]: pgmap v1747: 196 pgs: 196 active+clean; 582 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:35:11.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:11 smithi173 ceph-mon[33839]: osdmap e816: 8 total, 8 up, 8 in 2024-03-26T02:35:11.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:11 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/254221309' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pg_num": 12}]: dispatch 2024-03-26T02:35:11.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:11 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pg_num": 12}]: dispatch 2024-03-26T02:35:11.625 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:11.624+0000 7f3efb11d640 1 -- 172.21.15.72:0/3140409301 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f3ef410fc40 con 0x7f3ef4100e50 2024-03-26T02:35:11.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:11 smithi072 ceph-mon[41619]: pgmap v1747: 196 pgs: 196 active+clean; 582 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:35:11.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:11 smithi072 ceph-mon[41619]: osdmap e816: 8 total, 8 up, 8 in 2024-03-26T02:35:11.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:11 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/254221309' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pg_num": 12}]: dispatch 2024-03-26T02:35:11.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:11 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pg_num": 12}]: dispatch 2024-03-26T02:35:11.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:11 smithi072 ceph-mon[30177]: pgmap v1747: 196 pgs: 196 active+clean; 582 KiB data, 889 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:35:11.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:11 smithi072 ceph-mon[30177]: osdmap e816: 8 total, 8 up, 8 in 2024-03-26T02:35:11.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:11 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/254221309' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pg_num": 12}]: dispatch 2024-03-26T02:35:11.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:11 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pg_num": 12}]: dispatch 2024-03-26T02:35:12.249 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:12.248+0000 7f3ef1ffb640 1 -- 172.21.15.72:0/3140409301 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool de5f38de-babb-481b-9bb5-ced33138378b v818) v1 ==== 223+0+0 (secure 0 0 0) 0x7f3ee005a560 con 0x7f3ef4100e50 2024-03-26T02:35:12.360 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:12.359+0000 7f3efb11d640 1 -- 172.21.15.72:0/3140409301 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f3ef410fdd0 con 0x7f3ef4100e50 2024-03-26T02:35:12.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:35:12 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:35:12.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pg_num": 12}]': finished 2024-03-26T02:35:12.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:12 smithi173 ceph-mon[33839]: osdmap e817: 8 total, 8 up, 8 in 2024-03-26T02:35:12.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/254221309' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pg_num": 12}]: dispatch 2024-03-26T02:35:12.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pg_num": 12}]: dispatch 2024-03-26T02:35:12.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:12 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3140409301' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T02:35:12.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:12 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T02:35:12.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pg_num": 12}]': finished 2024-03-26T02:35:12.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:12 smithi072 ceph-mon[30177]: osdmap e817: 8 total, 8 up, 8 in 2024-03-26T02:35:12.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/254221309' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pg_num": 12}]: dispatch 2024-03-26T02:35:12.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pg_num": 12}]: dispatch 2024-03-26T02:35:12.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:12 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3140409301' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T02:35:12.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:12 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T02:35:12.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pg_num": 12}]': finished 2024-03-26T02:35:12.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:12 smithi072 ceph-mon[41619]: osdmap e817: 8 total, 8 up, 8 in 2024-03-26T02:35:12.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/254221309' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pg_num": 12}]: dispatch 2024-03-26T02:35:12.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pg_num": 12}]: dispatch 2024-03-26T02:35:12.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:12 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3140409301' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T02:35:12.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:12 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T02:35:13.250 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.249+0000 7f3ef1ffb640 1 -- 172.21.15.72:0/3140409301 <== mon.2 v2:172.21.15.72:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool de5f38de-babb-481b-9bb5-ced33138378b v819) v1 ==== 223+0+0 (secure 0 0 0) 0x7f3ee005a740 con 0x7f3ef4100e50 2024-03-26T02:35:13.250 INFO:tasks.workunit.client.0.smithi072.stderr:set-quota max_objects = 10 for pool de5f38de-babb-481b-9bb5-ced33138378b 2024-03-26T02:35:13.254 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.252+0000 7f3efb11d640 1 -- 172.21.15.72:0/3140409301 >> v2:172.21.15.72:6800/3246690770 conn(0x7f3ee406d9f0 msgr2=0x7f3ee406feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:35:13.254 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.252+0000 7f3efb11d640 1 --2- 172.21.15.72:0/3140409301 >> v2:172.21.15.72:6800/3246690770 conn(0x7f3ee406d9f0 0x7f3ee406feb0 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 crypto rx=0x7f3ee80323d0 tx=0x7f3ee8035f30 comp rx=0 tx=0).stop 2024-03-26T02:35:13.254 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.252+0000 7f3efb11d640 1 -- 172.21.15.72:0/3140409301 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3ef4100e50 msgr2=0x7f3ef41977d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:35:13.254 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.252+0000 7f3efb11d640 1 --2- 172.21.15.72:0/3140409301 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3ef4100e50 0x7f3ef41977d0 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7f3ee000c920 tx=0x7f3ee000cde0 comp rx=0 tx=0).stop 2024-03-26T02:35:13.254 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.252+0000 7f3efb11d640 1 -- 172.21.15.72:0/3140409301 shutdown_connections 2024-03-26T02:35:13.254 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.252+0000 7f3efb11d640 1 --2- 172.21.15.72:0/3140409301 >> v2:172.21.15.72:6800/3246690770 conn(0x7f3ee406d9f0 0x7f3ee406feb0 unknown :-1 s=CLOSED pgs=3369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:13.254 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.252+0000 7f3efb11d640 1 --2- 172.21.15.72:0/3140409301 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3ef4106550 0x7f3ef419bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:13.254 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.252+0000 7f3efb11d640 1 --2- 172.21.15.72:0/3140409301 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3ef4100e50 0x7f3ef41977d0 unknown :-1 s=CLOSED pgs=2532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:13.254 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.252+0000 7f3efb11d640 1 --2- 172.21.15.72:0/3140409301 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3ef4100490 0x7f3ef4197290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:13.254 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.252+0000 7f3efb11d640 1 -- 172.21.15.72:0/3140409301 >> 172.21.15.72:0/3140409301 conn(0x7f3ef4077580 msgr2=0x7f3ef4069cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:35:13.254 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.253+0000 7f3efb11d640 1 -- 172.21.15.72:0/3140409301 shutdown_connections 2024-03-26T02:35:13.254 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.253+0000 7f3efb11d640 1 -- 172.21.15.72:0/3140409301 wait complete. 2024-03-26T02:35:13.265 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph osd pool application enable de5f38de-babb-481b-9bb5-ced33138378b rados 2024-03-26T02:35:13.352 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.351+0000 7f3964eda640 1 -- 172.21.15.72:0/1189861473 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f39601064f0 msgr2=0x7f39601068d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:35:13.353 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.351+0000 7f3964eda640 1 --2- 172.21.15.72:0/1189861473 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f39601064f0 0x7f39601068d0 secure :-1 s=READY pgs=2533 cs=0 l=1 rev1=1 crypto rx=0x7f3950009cd0 tx=0x7f395002c0f0 comp rx=0 tx=0).stop 2024-03-26T02:35:13.353 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.351+0000 7f3964eda640 1 -- 172.21.15.72:0/1189861473 shutdown_connections 2024-03-26T02:35:13.353 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.351+0000 7f3964eda640 1 --2- 172.21.15.72:0/1189861473 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f3960100e10 0x7f396010d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:13.353 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.351+0000 7f3964eda640 1 --2- 172.21.15.72:0/1189861473 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3960100450 0x7f39601008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:13.353 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.351+0000 7f3964eda640 1 --2- 172.21.15.72:0/1189861473 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f39601064f0 0x7f39601068d0 unknown :-1 s=CLOSED pgs=2533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:13.353 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.351+0000 7f3964eda640 1 -- 172.21.15.72:0/1189861473 >> 172.21.15.72:0/1189861473 conn(0x7f39600775b0 msgr2=0x7f39600779c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:35:13.353 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.352+0000 7f3964eda640 1 -- 172.21.15.72:0/1189861473 shutdown_connections 2024-03-26T02:35:13.353 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.352+0000 7f3964eda640 1 -- 172.21.15.72:0/1189861473 wait complete. 2024-03-26T02:35:13.355 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.352+0000 7f3964eda640 1 Processor -- start 2024-03-26T02:35:13.355 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.352+0000 7f3964eda640 1 -- start start 2024-03-26T02:35:13.355 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.352+0000 7f3964eda640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3960100450 0x7f3960196310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:35:13.355 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.352+0000 7f3964eda640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3960100e10 0x7f3960196850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:35:13.356 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.352+0000 7f3964eda640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39601064f0 0x7f396019abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:35:13.356 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.352+0000 7f3964eda640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396019b1d0 con 0x7f39601064f0 2024-03-26T02:35:13.356 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.352+0000 7f3964eda640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396019b340 con 0x7f3960100e10 2024-03-26T02:35:13.356 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.352+0000 7f3964eda640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f396019b500 con 0x7f3960100450 2024-03-26T02:35:13.356 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.353+0000 7f395e59c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3960100450 0x7f3960196310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:35:13.356 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.353+0000 7f395e59c640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3960100450 0x7f3960196310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:59124/0 (socket says 172.21.15.72:59124) 2024-03-26T02:35:13.356 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.353+0000 7f395ed9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39601064f0 0x7f396019abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:35:13.356 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.353+0000 7f395e59c640 1 -- 172.21.15.72:0/252476674 learned_addr learned my addr 172.21.15.72:0/252476674 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:35:13.356 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.353+0000 7f395dd9b640 1 --2- 172.21.15.72:0/252476674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3960100e10 0x7f3960196850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:35:13.356 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.353+0000 7f395e59c640 1 -- 172.21.15.72:0/252476674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3960100e10 msgr2=0x7f3960196850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:35:13.356 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.353+0000 7f395e59c640 1 --2- 172.21.15.72:0/252476674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3960100e10 0x7f3960196850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:13.356 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.353+0000 7f395e59c640 1 -- 172.21.15.72:0/252476674 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39601064f0 msgr2=0x7f396019abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:35:13.357 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.353+0000 7f395e59c640 1 --2- 172.21.15.72:0/252476674 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39601064f0 0x7f396019abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:13.357 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.353+0000 7f395e59c640 1 -- 172.21.15.72:0/252476674 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3958009490 con 0x7f3960100450 2024-03-26T02:35:13.357 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.353+0000 7f395ed9d640 1 --2- 172.21.15.72:0/252476674 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39601064f0 0x7f396019abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:35:13.357 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.353+0000 7f395e59c640 1 --2- 172.21.15.72:0/252476674 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3960100450 0x7f3960196310 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7f3950004af0 tx=0x7f39500346e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:35:13.357 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.353+0000 7f394f7fe640 1 -- 172.21.15.72:0/252476674 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3950032980 con 0x7f3960100450 2024-03-26T02:35:13.357 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.353+0000 7f394f7fe640 1 -- 172.21.15.72:0/252476674 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f3950032ae0 con 0x7f3960100450 2024-03-26T02:35:13.357 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.353+0000 7f3964eda640 1 -- 172.21.15.72:0/252476674 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3950009900 con 0x7f3960100450 2024-03-26T02:35:13.357 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.353+0000 7f394f7fe640 1 -- 172.21.15.72:0/252476674 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39500042a0 con 0x7f3960100450 2024-03-26T02:35:13.357 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.353+0000 7f3964eda640 1 -- 172.21.15.72:0/252476674 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39600688c0 con 0x7f3960100450 2024-03-26T02:35:13.357 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.354+0000 7f3964eda640 1 -- 172.21.15.72:0/252476674 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3960101b90 con 0x7f3960100450 2024-03-26T02:35:13.358 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.355+0000 7f394f7fe640 1 -- 172.21.15.72:0/252476674 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7f395003b070 con 0x7f3960100450 2024-03-26T02:35:13.358 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.355+0000 7f394f7fe640 1 --2- 172.21.15.72:0/252476674 >> v2:172.21.15.72:6800/3246690770 conn(0x7f394806d9f0 0x7f394806feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:35:13.358 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.355+0000 7f394f7fe640 1 -- 172.21.15.72:0/252476674 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(819..819 src has 255..819) v4 ==== 7833+0+0 (secure 0 0 0) 0x7f39500b1c70 con 0x7f3960100450 2024-03-26T02:35:13.358 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.355+0000 7f395dd9b640 1 --2- 172.21.15.72:0/252476674 >> v2:172.21.15.72:6800/3246690770 conn(0x7f394806d9f0 0x7f394806feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:35:13.358 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.356+0000 7f395dd9b640 1 --2- 172.21.15.72:0/252476674 >> v2:172.21.15.72:6800/3246690770 conn(0x7f394806d9f0 0x7f394806feb0 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 crypto rx=0x7f3960069eb0 tx=0x7f3954008040 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:35:13.361 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.360+0000 7f394f7fe640 1 -- 172.21.15.72:0/252476674 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f395007ea70 con 0x7f3960100450 2024-03-26T02:35:13.538 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:13.537+0000 7f3964eda640 1 -- 172.21.15.72:0/252476674 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "app": "rados"} v 0) v1 -- 0x7f3960197660 con 0x7f3960100450 2024-03-26T02:35:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:13 smithi173 ceph-mon[33839]: pgmap v1750: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 831 B/s rd, 0 op/s 2024-03-26T02:35:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:13 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:13 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "10"}]': finished 2024-03-26T02:35:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:13 smithi173 ceph-mon[33839]: osdmap e818: 8 total, 8 up, 8 in 2024-03-26T02:35:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:13 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3140409301' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T02:35:13.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:13 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T02:35:13.620 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:13 smithi072 ceph-mon[30177]: pgmap v1750: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 831 B/s rd, 0 op/s 2024-03-26T02:35:13.620 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:13 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:13.620 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:13 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "10"}]': finished 2024-03-26T02:35:13.620 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:13 smithi072 ceph-mon[30177]: osdmap e818: 8 total, 8 up, 8 in 2024-03-26T02:35:13.620 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:13 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3140409301' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T02:35:13.620 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:13 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T02:35:13.621 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:13 smithi072 ceph-mon[41619]: pgmap v1750: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 831 B/s rd, 0 op/s 2024-03-26T02:35:13.621 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:13 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:13.621 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:13 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "10"}]': finished 2024-03-26T02:35:13.621 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:13 smithi072 ceph-mon[41619]: osdmap e818: 8 total, 8 up, 8 in 2024-03-26T02:35:13.621 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:13 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3140409301' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T02:35:13.621 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:13 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T02:35:14.264 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:14.263+0000 7f394f7fe640 1 -- 172.21.15.72:0/252476674 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "app": "rados"}]=0 enabled application 'rados' on pool 'de5f38de-babb-481b-9bb5-ced33138378b' v820) v1 ==== 213+0+0 (secure 0 0 0) 0x7f395007e600 con 0x7f3960100450 2024-03-26T02:35:14.374 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:14.373+0000 7f3964eda640 1 -- 172.21.15.72:0/252476674 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "app": "rados"} v 0) v1 -- 0x7f396010fe30 con 0x7f3960100450 2024-03-26T02:35:14.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:14 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "10"}]': finished 2024-03-26T02:35:14.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:14 smithi173 ceph-mon[33839]: osdmap e819: 8 total, 8 up, 8 in 2024-03-26T02:35:14.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:14 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/252476674' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "app": "rados"}]: dispatch 2024-03-26T02:35:14.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:14 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "app": "rados"}]: dispatch 2024-03-26T02:35:14.640 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:14 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "10"}]': finished 2024-03-26T02:35:14.640 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:14 smithi072 ceph-mon[30177]: osdmap e819: 8 total, 8 up, 8 in 2024-03-26T02:35:14.640 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:14 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/252476674' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "app": "rados"}]: dispatch 2024-03-26T02:35:14.640 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:14 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "app": "rados"}]: dispatch 2024-03-26T02:35:14.640 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:14 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "10"}]': finished 2024-03-26T02:35:14.640 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:14 smithi072 ceph-mon[41619]: osdmap e819: 8 total, 8 up, 8 in 2024-03-26T02:35:14.640 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:14 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/252476674' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "app": "rados"}]: dispatch 2024-03-26T02:35:14.640 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:14 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "app": "rados"}]: dispatch 2024-03-26T02:35:15.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:35:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:35:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:35:15.272 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:15.271+0000 7f394f7fe640 1 -- 172.21.15.72:0/252476674 <== mon.2 v2:172.21.15.72:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "app": "rados"}]=0 enabled application 'rados' on pool 'de5f38de-babb-481b-9bb5-ced33138378b' v821) v1 ==== 213+0+0 (secure 0 0 0) 0x7f395004c070 con 0x7f3960100450 2024-03-26T02:35:15.272 INFO:tasks.workunit.client.0.smithi072.stderr:enabled application 'rados' on pool 'de5f38de-babb-481b-9bb5-ced33138378b' 2024-03-26T02:35:15.275 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:15.274+0000 7f3964eda640 1 -- 172.21.15.72:0/252476674 >> v2:172.21.15.72:6800/3246690770 conn(0x7f394806d9f0 msgr2=0x7f394806feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:35:15.275 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:15.274+0000 7f3964eda640 1 --2- 172.21.15.72:0/252476674 >> v2:172.21.15.72:6800/3246690770 conn(0x7f394806d9f0 0x7f394806feb0 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 crypto rx=0x7f3960069eb0 tx=0x7f3954008040 comp rx=0 tx=0).stop 2024-03-26T02:35:15.275 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:15.274+0000 7f3964eda640 1 -- 172.21.15.72:0/252476674 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3960100450 msgr2=0x7f3960196310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:35:15.275 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:15.274+0000 7f3964eda640 1 --2- 172.21.15.72:0/252476674 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3960100450 0x7f3960196310 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7f3950004af0 tx=0x7f39500346e0 comp rx=0 tx=0).stop 2024-03-26T02:35:15.275 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:15.274+0000 7f3964eda640 1 -- 172.21.15.72:0/252476674 shutdown_connections 2024-03-26T02:35:15.276 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:15.274+0000 7f3964eda640 1 --2- 172.21.15.72:0/252476674 >> v2:172.21.15.72:6800/3246690770 conn(0x7f394806d9f0 0x7f394806feb0 unknown :-1 s=CLOSED pgs=3370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:15.276 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:15.274+0000 7f3964eda640 1 --2- 172.21.15.72:0/252476674 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f39601064f0 0x7f396019abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:15.276 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:15.274+0000 7f3964eda640 1 --2- 172.21.15.72:0/252476674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3960100e10 0x7f3960196850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:15.276 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:15.274+0000 7f3964eda640 1 --2- 172.21.15.72:0/252476674 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f3960100450 0x7f3960196310 unknown :-1 s=CLOSED pgs=2534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:15.276 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:15.274+0000 7f3964eda640 1 -- 172.21.15.72:0/252476674 >> 172.21.15.72:0/252476674 conn(0x7f39600775b0 msgr2=0x7f39600ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:35:15.276 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:15.275+0000 7f3964eda640 1 -- 172.21.15.72:0/252476674 shutdown_connections 2024-03-26T02:35:15.277 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:15.275+0000 7f3964eda640 1 -- 172.21.15.72:0/252476674 wait complete. 2024-03-26T02:35:15.286 INFO:tasks.workunit.client.0.smithi072.stderr:++ seq 1 10 2024-03-26T02:35:15.291 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in `seq 1 10` 2024-03-26T02:35:15.291 INFO:tasks.workunit.client.0.smithi072.stderr:+ rados -p de5f38de-babb-481b-9bb5-ced33138378b put obj1 /etc/passwd 2024-03-26T02:35:15.328 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in `seq 1 10` 2024-03-26T02:35:15.328 INFO:tasks.workunit.client.0.smithi072.stderr:+ rados -p de5f38de-babb-481b-9bb5-ced33138378b put obj2 /etc/passwd 2024-03-26T02:35:15.364 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in `seq 1 10` 2024-03-26T02:35:15.365 INFO:tasks.workunit.client.0.smithi072.stderr:+ rados -p de5f38de-babb-481b-9bb5-ced33138378b put obj3 /etc/passwd 2024-03-26T02:35:15.400 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in `seq 1 10` 2024-03-26T02:35:15.400 INFO:tasks.workunit.client.0.smithi072.stderr:+ rados -p de5f38de-babb-481b-9bb5-ced33138378b put obj4 /etc/passwd 2024-03-26T02:35:15.435 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in `seq 1 10` 2024-03-26T02:35:15.435 INFO:tasks.workunit.client.0.smithi072.stderr:+ rados -p de5f38de-babb-481b-9bb5-ced33138378b put obj5 /etc/passwd 2024-03-26T02:35:15.470 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in `seq 1 10` 2024-03-26T02:35:15.471 INFO:tasks.workunit.client.0.smithi072.stderr:+ rados -p de5f38de-babb-481b-9bb5-ced33138378b put obj6 /etc/passwd 2024-03-26T02:35:15.506 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in `seq 1 10` 2024-03-26T02:35:15.506 INFO:tasks.workunit.client.0.smithi072.stderr:+ rados -p de5f38de-babb-481b-9bb5-ced33138378b put obj7 /etc/passwd 2024-03-26T02:35:15.541 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in `seq 1 10` 2024-03-26T02:35:15.541 INFO:tasks.workunit.client.0.smithi072.stderr:+ rados -p de5f38de-babb-481b-9bb5-ced33138378b put obj8 /etc/passwd 2024-03-26T02:35:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:15 smithi173 ceph-mon[33839]: pgmap v1753: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "app": "rados"}]': finished 2024-03-26T02:35:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:15 smithi173 ceph-mon[33839]: osdmap e820: 8 total, 8 up, 8 in 2024-03-26T02:35:15.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:15 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/252476674' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "app": "rados"}]: dispatch 2024-03-26T02:35:15.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:15 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "app": "rados"}]: dispatch 2024-03-26T02:35:15.577 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in `seq 1 10` 2024-03-26T02:35:15.577 INFO:tasks.workunit.client.0.smithi072.stderr:+ rados -p de5f38de-babb-481b-9bb5-ced33138378b put obj9 /etc/passwd 2024-03-26T02:35:15.611 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in `seq 1 10` 2024-03-26T02:35:15.611 INFO:tasks.workunit.client.0.smithi072.stderr:+ rados -p de5f38de-babb-481b-9bb5-ced33138378b put obj10 /etc/passwd 2024-03-26T02:35:15.646 INFO:tasks.workunit.client.0.smithi072.stderr:+ sleep 30 2024-03-26T02:35:15.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:15 smithi072 ceph-mon[30177]: pgmap v1753: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:15.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "app": "rados"}]': finished 2024-03-26T02:35:15.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:15 smithi072 ceph-mon[30177]: osdmap e820: 8 total, 8 up, 8 in 2024-03-26T02:35:15.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:15 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/252476674' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "app": "rados"}]: dispatch 2024-03-26T02:35:15.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:15 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "app": "rados"}]: dispatch 2024-03-26T02:35:15.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:15 smithi072 ceph-mon[41619]: pgmap v1753: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:15.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "app": "rados"}]': finished 2024-03-26T02:35:15.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:15 smithi072 ceph-mon[41619]: osdmap e820: 8 total, 8 up, 8 in 2024-03-26T02:35:15.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:15 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/252476674' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "app": "rados"}]: dispatch 2024-03-26T02:35:15.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:15 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "app": "rados"}]: dispatch 2024-03-26T02:35:16.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:16 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "app": "rados"}]': finished 2024-03-26T02:35:16.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:16 smithi173 ceph-mon[33839]: osdmap e821: 8 total, 8 up, 8 in 2024-03-26T02:35:16.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:16 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "app": "rados"}]': finished 2024-03-26T02:35:16.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:16 smithi072 ceph-mon[30177]: osdmap e821: 8 total, 8 up, 8 in 2024-03-26T02:35:16.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:16 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "app": "rados"}]': finished 2024-03-26T02:35:16.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:16 smithi072 ceph-mon[41619]: osdmap e821: 8 total, 8 up, 8 in 2024-03-26T02:35:17.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:17 smithi173 ceph-mon[33839]: pgmap v1756: 176 pgs: 176 active+clean; 628 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-03-26T02:35:17.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:17 smithi173 ceph-mon[33839]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:35:17.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:17 smithi072 ceph-mon[30177]: pgmap v1756: 176 pgs: 176 active+clean; 628 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-03-26T02:35:17.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:17 smithi072 ceph-mon[30177]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:35:17.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:17 smithi072 ceph-mon[41619]: pgmap v1756: 176 pgs: 176 active+clean; 628 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-03-26T02:35:17.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:17 smithi072 ceph-mon[41619]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T02:35:19.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:19 smithi173 ceph-mon[33839]: pgmap v1757: 176 pgs: 176 active+clean; 628 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 864 B/s rd, 8.1 KiB/s wr, 1 op/s 2024-03-26T02:35:19.722 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:19 smithi072 ceph-mon[30177]: pgmap v1757: 176 pgs: 176 active+clean; 628 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 864 B/s rd, 8.1 KiB/s wr, 1 op/s 2024-03-26T02:35:19.723 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:19 smithi072 ceph-mon[41619]: pgmap v1757: 176 pgs: 176 active+clean; 628 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 864 B/s rd, 8.1 KiB/s wr, 1 op/s 2024-03-26T02:35:20.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:20 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:35:20.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:20 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:35:20.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:20 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:35:21.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:21 smithi173 ceph-mon[33839]: pgmap v1758: 176 pgs: 176 active+clean; 628 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 739 B/s rd, 6.9 KiB/s wr, 1 op/s 2024-03-26T02:35:21.606 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:21 smithi072 ceph-mon[30177]: pgmap v1758: 176 pgs: 176 active+clean; 628 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 739 B/s rd, 6.9 KiB/s wr, 1 op/s 2024-03-26T02:35:21.607 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:21 smithi072 ceph-mon[41619]: pgmap v1758: 176 pgs: 176 active+clean; 628 KiB data, 893 MiB used, 714 GiB / 715 GiB avail; 739 B/s rd, 6.9 KiB/s wr, 1 op/s 2024-03-26T02:35:22.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:35:22 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:35:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:23 smithi173 ceph-mon[33839]: pgmap v1759: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-03-26T02:35:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:23 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:23 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:35:23.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:23 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:35:23.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:23 smithi072 ceph-mon[41619]: pgmap v1759: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-03-26T02:35:23.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:23 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:23.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:23 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:35:23.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:23 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:35:23.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:23 smithi072 ceph-mon[30177]: pgmap v1759: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 15 KiB/s wr, 2 op/s 2024-03-26T02:35:23.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:23 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:23.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:23 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:35:23.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:23 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:35:24.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:24 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:35:24.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:24 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:35:24.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:24 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:35:24.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:24 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:35:24.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:24 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:35:24.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:24 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:35:24.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:24 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:35:24.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:24 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:35:24.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:24 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:35:25.295 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:35:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:35:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:35:25.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:25 smithi173 ceph-mon[33839]: pgmap v1760: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-03-26T02:35:25.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:25 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:35:25.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:25 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:35:25.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:25 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:35:25.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:25 smithi072 ceph-mon[30177]: pgmap v1760: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-03-26T02:35:25.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:25 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:35:25.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:25 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:35:25.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:25 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:35:25.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:25 smithi072 ceph-mon[41619]: pgmap v1760: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 12 KiB/s wr, 2 op/s 2024-03-26T02:35:25.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:25 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:35:25.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:25 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:35:25.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:25 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:35:26.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:26 smithi173 ceph-mon[33839]: pool 'de5f38de-babb-481b-9bb5-ced33138378b' is full (reached quota's max_objects: 10) 2024-03-26T02:35:26.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:26 smithi173 ceph-mon[33839]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T02:35:26.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:26 smithi173 ceph-mon[33839]: osdmap e822: 8 total, 8 up, 8 in 2024-03-26T02:35:26.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:26 smithi072 ceph-mon[41619]: pool 'de5f38de-babb-481b-9bb5-ced33138378b' is full (reached quota's max_objects: 10) 2024-03-26T02:35:26.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:26 smithi072 ceph-mon[41619]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T02:35:26.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:26 smithi072 ceph-mon[41619]: osdmap e822: 8 total, 8 up, 8 in 2024-03-26T02:35:26.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:26 smithi072 ceph-mon[30177]: pool 'de5f38de-babb-481b-9bb5-ced33138378b' is full (reached quota's max_objects: 10) 2024-03-26T02:35:26.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:26 smithi072 ceph-mon[30177]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T02:35:26.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:26 smithi072 ceph-mon[30177]: osdmap e822: 8 total, 8 up, 8 in 2024-03-26T02:35:27.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:27 smithi173 ceph-mon[33839]: pgmap v1762: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 7.2 KiB/s wr, 1 op/s 2024-03-26T02:35:27.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:27 smithi072 ceph-mon[30177]: pgmap v1762: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 7.2 KiB/s wr, 1 op/s 2024-03-26T02:35:27.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:27 smithi072 ceph-mon[41619]: pgmap v1762: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 7.2 KiB/s wr, 1 op/s 2024-03-26T02:35:29.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:29 smithi173 ceph-mon[33839]: pgmap v1763: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 7.2 KiB/s wr, 1 op/s 2024-03-26T02:35:29.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:29 smithi072 ceph-mon[30177]: pgmap v1763: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 7.2 KiB/s wr, 1 op/s 2024-03-26T02:35:29.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:29 smithi072 ceph-mon[41619]: pgmap v1763: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 7.2 KiB/s wr, 1 op/s 2024-03-26T02:35:31.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:31 smithi173 ceph-mon[33839]: pgmap v1764: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 7.2 KiB/s wr, 1 op/s 2024-03-26T02:35:31.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:31 smithi072 ceph-mon[30177]: pgmap v1764: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 7.2 KiB/s wr, 1 op/s 2024-03-26T02:35:31.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:31 smithi072 ceph-mon[41619]: pgmap v1764: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 7.2 KiB/s wr, 1 op/s 2024-03-26T02:35:32.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:35:32 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:35:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:33 smithi173 ceph-mon[33839]: pgmap v1765: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:35:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:33 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:33.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:33 smithi072 ceph-mon[30177]: pgmap v1765: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:35:33.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:33 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:33.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:33 smithi072 ceph-mon[41619]: pgmap v1765: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:35:33.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:33 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:35.309 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:35:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:35:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:35:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:35 smithi173 ceph-mon[33839]: pgmap v1766: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:35:35.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:35 smithi072 ceph-mon[41619]: pgmap v1766: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:35:35.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:35 smithi072 ceph-mon[30177]: pgmap v1766: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:35:37.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:37 smithi173 ceph-mon[33839]: pgmap v1767: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2024-03-26T02:35:37.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:37 smithi072 ceph-mon[30177]: pgmap v1767: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2024-03-26T02:35:37.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:37 smithi072 ceph-mon[41619]: pgmap v1767: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2024-03-26T02:35:39.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:39 smithi072 ceph-mon[30177]: pgmap v1768: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:39.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:39 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:35:39.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:39 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:35:39.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:39 smithi072 ceph-mon[41619]: pgmap v1768: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:39.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:39 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:35:39.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:39 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:35:39.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:39 smithi173 ceph-mon[33839]: pgmap v1768: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:39.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:39 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:35:39.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:39 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:35:41.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:41 smithi072 ceph-mon[30177]: pgmap v1769: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:41.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:41 smithi072 ceph-mon[41619]: pgmap v1769: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:41.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:41 smithi173 ceph-mon[33839]: pgmap v1769: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:42.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:35:42 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:35:43.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:43 smithi072 ceph-mon[30177]: pgmap v1770: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:43.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:43 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:43.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:43 smithi072 ceph-mon[41619]: pgmap v1770: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:43.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:43 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:43.769 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:43 smithi173 ceph-mon[33839]: pgmap v1770: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:43.769 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:43 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:45.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:35:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:35:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:35:45.647 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=159323 2024-03-26T02:35:45.648 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph osd pool set-quota de5f38de-babb-481b-9bb5-ced33138378b max_objects 100 2024-03-26T02:35:45.648 INFO:tasks.workunit.client.0.smithi072.stderr:+ rados -p de5f38de-babb-481b-9bb5-ced33138378b put onemore /etc/passwd 2024-03-26T02:35:45.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:45 smithi072 ceph-mon[30177]: pgmap v1771: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:45.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:45 smithi072 ceph-mon[41619]: pgmap v1771: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:45.735 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.732+0000 7f7c30377640 1 -- 172.21.15.72:0/124946374 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7c28107640 msgr2=0x7f7c28107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:35:45.735 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.732+0000 7f7c30377640 1 --2- 172.21.15.72:0/124946374 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7c28107640 0x7f7c28107a20 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7f7c24009890 tx=0x7f7c2402bb70 comp rx=0 tx=0).stop 2024-03-26T02:35:45.735 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.732+0000 7f7c30377640 1 -- 172.21.15.72:0/124946374 shutdown_connections 2024-03-26T02:35:45.736 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.732+0000 7f7c30377640 1 --2- 172.21.15.72:0/124946374 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c28101f40 0x7f7c2810e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:45.736 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.732+0000 7f7c30377640 1 --2- 172.21.15.72:0/124946374 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7c28101580 0x7f7c28101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:45.736 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.732+0000 7f7c30377640 1 --2- 172.21.15.72:0/124946374 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7c28107640 0x7f7c28107a20 unknown :-1 s=CLOSED pgs=2558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:45.736 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.732+0000 7f7c30377640 1 -- 172.21.15.72:0/124946374 >> 172.21.15.72:0/124946374 conn(0x7f7c280fd090 msgr2=0x7f7c280ff4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:35:45.736 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.732+0000 7f7c30377640 1 -- 172.21.15.72:0/124946374 shutdown_connections 2024-03-26T02:35:45.736 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.732+0000 7f7c30377640 1 -- 172.21.15.72:0/124946374 wait complete. 2024-03-26T02:35:45.736 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.733+0000 7f7c30377640 1 Processor -- start 2024-03-26T02:35:45.736 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.733+0000 7f7c30377640 1 -- start start 2024-03-26T02:35:45.736 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.733+0000 7f7c30377640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7c28101580 0x7f7c28073890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:35:45.736 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.733+0000 7f7c30377640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7c28101f40 0x7f7c28073dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:35:45.736 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.733+0000 7f7c30377640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c28107640 0x7f7c28074310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:35:45.736 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.733+0000 7f7c30377640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c281a55f0 con 0x7f7c28101f40 2024-03-26T02:35:45.736 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.733+0000 7f7c30377640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c281a5760 con 0x7f7c28107640 2024-03-26T02:35:45.737 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.733+0000 7f7c30377640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c281a5920 con 0x7f7c28101580 2024-03-26T02:35:45.737 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.733+0000 7f7c2e113640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7c28101580 0x7f7c28073890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:35:45.737 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.733+0000 7f7c2d912640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7c28101f40 0x7f7c28073dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:35:45.737 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.733+0000 7f7c2e113640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7c28101580 0x7f7c28073890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:32944/0 (socket says 172.21.15.72:32944) 2024-03-26T02:35:45.737 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.733+0000 7f7c2e113640 1 -- 172.21.15.72:0/3066103712 learned_addr learned my addr 172.21.15.72:0/3066103712 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:35:45.737 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.733+0000 7f7c2e914640 1 --2- 172.21.15.72:0/3066103712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c28107640 0x7f7c28074310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:35:45.737 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.734+0000 7f7c2e113640 1 -- 172.21.15.72:0/3066103712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c28107640 msgr2=0x7f7c28074310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:35:45.737 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.734+0000 7f7c2e113640 1 --2- 172.21.15.72:0/3066103712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c28107640 0x7f7c28074310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:45.737 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.734+0000 7f7c2e113640 1 -- 172.21.15.72:0/3066103712 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7c28101f40 msgr2=0x7f7c28073dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:35:45.737 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.734+0000 7f7c2e113640 1 --2- 172.21.15.72:0/3066103712 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7c28101f40 0x7f7c28073dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:45.737 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.734+0000 7f7c2e113640 1 -- 172.21.15.72:0/3066103712 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c18009560 con 0x7f7c28101580 2024-03-26T02:35:45.737 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.734+0000 7f7c2d912640 1 --2- 172.21.15.72:0/3066103712 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7c28101f40 0x7f7c28073dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:35:45.737 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.734+0000 7f7c2e113640 1 --2- 172.21.15.72:0/3066103712 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7c28101580 0x7f7c28073890 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7f7c24009860 tx=0x7f7c24034670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:35:45.737 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.734+0000 7f7c1f7fe640 1 -- 172.21.15.72:0/3066103712 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c2402c710 con 0x7f7c28101580 2024-03-26T02:35:45.737 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.734+0000 7f7c1f7fe640 1 -- 172.21.15.72:0/3066103712 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f7c2402ce90 con 0x7f7c28101580 2024-03-26T02:35:45.738 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.734+0000 7f7c30377640 1 -- 172.21.15.72:0/3066103712 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c24009490 con 0x7f7c28101580 2024-03-26T02:35:45.738 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.734+0000 7f7c1f7fe640 1 -- 172.21.15.72:0/3066103712 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c240324a0 con 0x7f7c28101580 2024-03-26T02:35:45.738 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.734+0000 7f7c30377640 1 -- 172.21.15.72:0/3066103712 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c281a5ec0 con 0x7f7c28101580 2024-03-26T02:35:45.739 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.735+0000 7f7c30377640 1 -- 172.21.15.72:0/3066103712 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c280688c0 con 0x7f7c28101580 2024-03-26T02:35:45.739 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.736+0000 7f7c1f7fe640 1 -- 172.21.15.72:0/3066103712 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7f7c2403b040 con 0x7f7c28101580 2024-03-26T02:35:45.739 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.736+0000 7f7c1f7fe640 1 --2- 172.21.15.72:0/3066103712 >> v2:172.21.15.72:6800/3246690770 conn(0x7f7c1406da20 0x7f7c1406fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:35:45.739 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.736+0000 7f7c2d912640 1 --2- 172.21.15.72:0/3066103712 >> v2:172.21.15.72:6800/3246690770 conn(0x7f7c1406da20 0x7f7c1406fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:35:45.739 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.736+0000 7f7c1f7fe640 1 -- 172.21.15.72:0/3066103712 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(822..822 src has 255..822) v4 ==== 7846+0+0 (secure 0 0 0) 0x7f7c240af520 con 0x7f7c28101580 2024-03-26T02:35:45.739 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.736+0000 7f7c1f7fe640 1 -- 172.21.15.72:0/3066103712 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=823}) v3 -- 0x7f7c14078f90 con 0x7f7c28101580 2024-03-26T02:35:45.739 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.737+0000 7f7c2d912640 1 --2- 172.21.15.72:0/3066103712 >> v2:172.21.15.72:6800/3246690770 conn(0x7f7c1406da20 0x7f7c1406fee0 secure :-1 s=READY pgs=3382 cs=0 l=1 rev1=1 crypto rx=0x7f7c280752d0 tx=0x7f7c18009260 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:35:45.742 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.741+0000 7f7c1f7fe640 1 -- 172.21.15.72:0/3066103712 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f7c2407c3c0 con 0x7f7c28101580 2024-03-26T02:35:45.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:45 smithi173 ceph-mon[33839]: pgmap v1771: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:45.916 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:45.915+0000 7f7c30377640 1 -- 172.21.15.72:0/3066103712 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "100"} v 0) v1 -- 0x7f7c28110d60 con 0x7f7c28101580 2024-03-26T02:35:46.345 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:46.344+0000 7f7c1f7fe640 1 -- 172.21.15.72:0/3066103712 <== mon.2 v2:172.21.15.72:3301/0 7 ==== osd_map(823..823 src has 255..823) v4 ==== 628+0+0 (secure 0 0 0) 0x7f7c2409c0c0 con 0x7f7c28101580 2024-03-26T02:35:46.345 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:46.344+0000 7f7c1f7fe640 1 -- 172.21.15.72:0/3066103712 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=824}) v3 -- 0x7f7c140798c0 con 0x7f7c28101580 2024-03-26T02:35:46.346 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:46.346+0000 7f7c1f7fe640 1 -- 172.21.15.72:0/3066103712 <== mon.2 v2:172.21.15.72:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool de5f38de-babb-481b-9bb5-ced33138378b v823) v1 ==== 225+0+0 (secure 0 0 0) 0x7f7c240390d0 con 0x7f7c28101580 2024-03-26T02:35:46.456 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:46.454+0000 7f7c30377640 1 -- 172.21.15.72:0/3066103712 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "100"} v 0) v1 -- 0x7f7c281a5ab0 con 0x7f7c28101580 2024-03-26T02:35:46.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:46 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3066103712' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T02:35:46.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:46 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T02:35:46.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:46 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3066103712' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T02:35:46.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:46 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T02:35:46.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:46 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3066103712' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T02:35:46.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:46 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T02:35:47.347 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:47.346+0000 7f7c1f7fe640 1 -- 172.21.15.72:0/3066103712 <== mon.2 v2:172.21.15.72:3301/0 9 ==== osd_map(824..824 src has 255..824) v4 ==== 628+0+0 (secure 0 0 0) 0x7f7c24080690 con 0x7f7c28101580 2024-03-26T02:35:47.347 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:47.346+0000 7f7c1f7fe640 1 -- 172.21.15.72:0/3066103712 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=825}) v3 -- 0x7f7c14079cf0 con 0x7f7c28101580 2024-03-26T02:35:47.351 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:47.350+0000 7f7c1f7fe640 1 -- 172.21.15.72:0/3066103712 <== mon.2 v2:172.21.15.72:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool de5f38de-babb-481b-9bb5-ced33138378b v824) v1 ==== 225+0+0 (secure 0 0 0) 0x7f7c2404b020 con 0x7f7c28101580 2024-03-26T02:35:47.351 INFO:tasks.workunit.client.0.smithi072.stderr:set-quota max_objects = 100 for pool de5f38de-babb-481b-9bb5-ced33138378b 2024-03-26T02:35:47.355 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:47.354+0000 7f7c30377640 1 -- 172.21.15.72:0/3066103712 >> v2:172.21.15.72:6800/3246690770 conn(0x7f7c1406da20 msgr2=0x7f7c1406fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:35:47.355 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:47.354+0000 7f7c30377640 1 --2- 172.21.15.72:0/3066103712 >> v2:172.21.15.72:6800/3246690770 conn(0x7f7c1406da20 0x7f7c1406fee0 secure :-1 s=READY pgs=3382 cs=0 l=1 rev1=1 crypto rx=0x7f7c280752d0 tx=0x7f7c18009260 comp rx=0 tx=0).stop 2024-03-26T02:35:47.355 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:47.354+0000 7f7c30377640 1 -- 172.21.15.72:0/3066103712 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7c28101580 msgr2=0x7f7c28073890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:35:47.355 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:47.354+0000 7f7c30377640 1 --2- 172.21.15.72:0/3066103712 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7c28101580 0x7f7c28073890 secure :-1 s=READY pgs=2559 cs=0 l=1 rev1=1 crypto rx=0x7f7c24009860 tx=0x7f7c24034670 comp rx=0 tx=0).stop 2024-03-26T02:35:47.356 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:47.354+0000 7f7c30377640 1 -- 172.21.15.72:0/3066103712 shutdown_connections 2024-03-26T02:35:47.356 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:47.354+0000 7f7c30377640 1 --2- 172.21.15.72:0/3066103712 >> v2:172.21.15.72:6800/3246690770 conn(0x7f7c1406da20 0x7f7c1406fee0 unknown :-1 s=CLOSED pgs=3382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:47.356 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:47.354+0000 7f7c30377640 1 --2- 172.21.15.72:0/3066103712 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c28107640 0x7f7c28074310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:47.356 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:47.354+0000 7f7c30377640 1 --2- 172.21.15.72:0/3066103712 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7c28101f40 0x7f7c28073dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:47.356 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:47.354+0000 7f7c30377640 1 --2- 172.21.15.72:0/3066103712 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7c28101580 0x7f7c28073890 unknown :-1 s=CLOSED pgs=2559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:47.356 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:47.354+0000 7f7c30377640 1 -- 172.21.15.72:0/3066103712 >> 172.21.15.72:0/3066103712 conn(0x7f7c280fd090 msgr2=0x7f7c28105130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:35:47.356 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:47.354+0000 7f7c30377640 1 -- 172.21.15.72:0/3066103712 shutdown_connections 2024-03-26T02:35:47.356 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:47.354+0000 7f7c30377640 1 -- 172.21.15.72:0/3066103712 wait complete. 2024-03-26T02:35:47.366 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 159323 2024-03-26T02:35:47.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:47 smithi072 ceph-mon[41619]: pgmap v1772: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:47.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:47 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "100"}]': finished 2024-03-26T02:35:47.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:47 smithi072 ceph-mon[41619]: osdmap e823: 8 total, 8 up, 8 in 2024-03-26T02:35:47.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:47 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3066103712' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T02:35:47.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:47 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T02:35:47.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:47 smithi072 ceph-mon[30177]: pgmap v1772: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:47.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:47 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "100"}]': finished 2024-03-26T02:35:47.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:47 smithi072 ceph-mon[30177]: osdmap e823: 8 total, 8 up, 8 in 2024-03-26T02:35:47.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:47 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3066103712' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T02:35:47.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:47 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T02:35:47.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:47 smithi173 ceph-mon[33839]: pgmap v1772: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:47.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:47 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "100"}]': finished 2024-03-26T02:35:47.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:47 smithi173 ceph-mon[33839]: osdmap e823: 8 total, 8 up, 8 in 2024-03-26T02:35:47.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:47 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3066103712' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T02:35:47.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:47 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T02:35:48.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:48 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "100"}]': finished 2024-03-26T02:35:48.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:48 smithi072 ceph-mon[30177]: osdmap e824: 8 total, 8 up, 8 in 2024-03-26T02:35:48.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:48 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "100"}]': finished 2024-03-26T02:35:48.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:48 smithi072 ceph-mon[41619]: osdmap e824: 8 total, 8 up, 8 in 2024-03-26T02:35:48.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:48 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "100"}]': finished 2024-03-26T02:35:48.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:48 smithi173 ceph-mon[33839]: osdmap e824: 8 total, 8 up, 8 in 2024-03-26T02:35:49.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:49 smithi072 ceph-mon[30177]: pgmap v1775: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:49.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:49 smithi072 ceph-mon[41619]: pgmap v1775: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:49.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:49 smithi173 ceph-mon[33839]: pgmap v1775: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:35:51.030 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 0 -ne 0 ']' 2024-03-26T02:35:51.030 INFO:tasks.workunit.client.0.smithi072.stderr:+ true 2024-03-26T02:35:51.030 INFO:tasks.workunit.client.0.smithi072.stderr:+ rados -p de5f38de-babb-481b-9bb5-ced33138378b put twomore /etc/passwd 2024-03-26T02:35:51.068 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph osd pool set-quota de5f38de-babb-481b-9bb5-ced33138378b max_bytes 100 2024-03-26T02:35:51.156 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.150+0000 7f83260bb640 1 -- 172.21.15.72:0/47030227 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8320102800 msgr2=0x7f8320102be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:35:51.156 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.150+0000 7f83260bb640 1 --2- 172.21.15.72:0/47030227 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8320102800 0x7f8320102be0 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7f8310009860 tx=0x7f831002bc00 comp rx=0 tx=0).stop 2024-03-26T02:35:51.156 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.150+0000 7f83260bb640 1 -- 172.21.15.72:0/47030227 shutdown_connections 2024-03-26T02:35:51.156 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.150+0000 7f83260bb640 1 --2- 172.21.15.72:0/47030227 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f832010bbd0 0x7f832010e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:51.156 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.150+0000 7f83260bb640 1 --2- 172.21.15.72:0/47030227 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8320103120 0x7f832010b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:51.157 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.150+0000 7f83260bb640 1 --2- 172.21.15.72:0/47030227 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f8320102800 0x7f8320102be0 unknown :-1 s=CLOSED pgs=2561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:51.157 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.150+0000 7f83260bb640 1 -- 172.21.15.72:0/47030227 >> 172.21.15.72:0/47030227 conn(0x7f8320077580 msgr2=0x7f8320077990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:35:51.157 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.150+0000 7f83260bb640 1 -- 172.21.15.72:0/47030227 shutdown_connections 2024-03-26T02:35:51.157 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.150+0000 7f83260bb640 1 -- 172.21.15.72:0/47030227 wait complete. 2024-03-26T02:35:51.157 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.151+0000 7f83260bb640 1 Processor -- start 2024-03-26T02:35:51.157 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.151+0000 7f83260bb640 1 -- start start 2024-03-26T02:35:51.157 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.151+0000 7f83260bb640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8320102800 0x7f8320197310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:35:51.157 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.151+0000 7f83260bb640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8320103120 0x7f8320197850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:35:51.157 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.151+0000 7f83260bb640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f832010bbd0 0x7f832019bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:35:51.157 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.151+0000 7f83260bb640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832019c1d0 con 0x7f8320103120 2024-03-26T02:35:51.158 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.151+0000 7f83260bb640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832019c340 con 0x7f8320102800 2024-03-26T02:35:51.158 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.151+0000 7f83260bb640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f832019c500 con 0x7f832010bbd0 2024-03-26T02:35:51.158 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.151+0000 7f831ffff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f832010bbd0 0x7f832019bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:35:51.158 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.151+0000 7f831effd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8320103120 0x7f8320197850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:35:51.158 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.151+0000 7f831ffff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f832010bbd0 0x7f832019bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:32992/0 (socket says 172.21.15.72:32992) 2024-03-26T02:35:51.158 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.151+0000 7f831ffff640 1 -- 172.21.15.72:0/626224747 learned_addr learned my addr 172.21.15.72:0/626224747 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:35:51.158 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.151+0000 7f831f7fe640 1 --2- 172.21.15.72:0/626224747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8320102800 0x7f8320197310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:35:51.159 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.151+0000 7f831ffff640 1 -- 172.21.15.72:0/626224747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8320102800 msgr2=0x7f8320197310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:35:51.159 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.152+0000 7f831ffff640 1 --2- 172.21.15.72:0/626224747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8320102800 0x7f8320197310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:51.159 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.152+0000 7f831ffff640 1 -- 172.21.15.72:0/626224747 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8320103120 msgr2=0x7f8320197850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:35:51.159 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.152+0000 7f831ffff640 1 --2- 172.21.15.72:0/626224747 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8320103120 0x7f8320197850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:51.159 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.152+0000 7f831ffff640 1 -- 172.21.15.72:0/626224747 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8318009900 con 0x7f832010bbd0 2024-03-26T02:35:51.159 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.152+0000 7f831f7fe640 1 --2- 172.21.15.72:0/626224747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8320102800 0x7f8320197310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T02:35:51.159 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.152+0000 7f831effd640 1 --2- 172.21.15.72:0/626224747 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8320103120 0x7f8320197850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:35:51.159 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.152+0000 7f831ffff640 1 --2- 172.21.15.72:0/626224747 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f832010bbd0 0x7f832019bbd0 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7f831800c3f0 tx=0x7f831800c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:35:51.159 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.152+0000 7f831cff9640 1 -- 172.21.15.72:0/626224747 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8318009be0 con 0x7f832010bbd0 2024-03-26T02:35:51.159 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.152+0000 7f83260bb640 1 -- 172.21.15.72:0/626224747 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8310009490 con 0x7f832010bbd0 2024-03-26T02:35:51.159 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.152+0000 7f831cff9640 1 -- 172.21.15.72:0/626224747 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f8318003e20 con 0x7f832010bbd0 2024-03-26T02:35:51.160 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.152+0000 7f831cff9640 1 -- 172.21.15.72:0/626224747 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8318004280 con 0x7f832010bbd0 2024-03-26T02:35:51.160 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.152+0000 7f83260bb640 1 -- 172.21.15.72:0/626224747 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8320100380 con 0x7f832010bbd0 2024-03-26T02:35:51.160 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.153+0000 7f83260bb640 1 -- 172.21.15.72:0/626224747 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83200688c0 con 0x7f832010bbd0 2024-03-26T02:35:51.160 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.153+0000 7f831cff9640 1 -- 172.21.15.72:0/626224747 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7f831800a040 con 0x7f832010bbd0 2024-03-26T02:35:51.160 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.154+0000 7f831cff9640 1 --2- 172.21.15.72:0/626224747 >> v2:172.21.15.72:6800/3246690770 conn(0x7f830806d9f0 0x7f830806feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:35:51.160 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.154+0000 7f831cff9640 1 -- 172.21.15.72:0/626224747 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(825..825 src has 255..825) v4 ==== 7846+0+0 (secure 0 0 0) 0x7f831808da80 con 0x7f832010bbd0 2024-03-26T02:35:51.160 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.154+0000 7f831f7fe640 1 --2- 172.21.15.72:0/626224747 >> v2:172.21.15.72:6800/3246690770 conn(0x7f830806d9f0 0x7f830806feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:35:51.160 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.154+0000 7f831f7fe640 1 --2- 172.21.15.72:0/626224747 >> v2:172.21.15.72:6800/3246690770 conn(0x7f830806d9f0 0x7f830806feb0 secure :-1 s=READY pgs=3384 cs=0 l=1 rev1=1 crypto rx=0x7f8310005b60 tx=0x7f8310000bc0 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:35:51.161 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.158+0000 7f831cff9640 1 -- 172.21.15.72:0/626224747 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f831805a870 con 0x7f832010bbd0 2024-03-26T02:35:51.335 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:51.333+0000 7f83260bb640 1 -- 172.21.15.72:0/626224747 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "100"} v 0) v1 -- 0x7f8320110d90 con 0x7f832010bbd0 2024-03-26T02:35:51.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:51 smithi072 ceph-mon[30177]: pgmap v1776: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:35:51.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:51 smithi072 ceph-mon[30177]: pool 'de5f38de-babb-481b-9bb5-ced33138378b' no longer out of quota; removing NO_QUOTA flag 2024-03-26T02:35:51.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:51 smithi072 ceph-mon[30177]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T02:35:51.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:51 smithi072 ceph-mon[30177]: osdmap e825: 8 total, 8 up, 8 in 2024-03-26T02:35:51.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:51 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/626224747' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T02:35:51.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:51 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T02:35:51.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:51 smithi072 ceph-mon[41619]: pgmap v1776: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:35:51.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:51 smithi072 ceph-mon[41619]: pool 'de5f38de-babb-481b-9bb5-ced33138378b' no longer out of quota; removing NO_QUOTA flag 2024-03-26T02:35:51.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:51 smithi072 ceph-mon[41619]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T02:35:51.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:51 smithi072 ceph-mon[41619]: osdmap e825: 8 total, 8 up, 8 in 2024-03-26T02:35:51.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:51 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/626224747' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T02:35:51.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:51 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T02:35:51.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:51 smithi173 ceph-mon[33839]: pgmap v1776: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:35:51.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:51 smithi173 ceph-mon[33839]: pool 'de5f38de-babb-481b-9bb5-ced33138378b' no longer out of quota; removing NO_QUOTA flag 2024-03-26T02:35:51.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:51 smithi173 ceph-mon[33839]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T02:35:51.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:51 smithi173 ceph-mon[33839]: osdmap e825: 8 total, 8 up, 8 in 2024-03-26T02:35:51.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:51 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/626224747' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T02:35:51.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:51 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T02:35:52.025 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:52.023+0000 7f831cff9640 1 -- 172.21.15.72:0/626224747 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool de5f38de-babb-481b-9bb5-ced33138378b v826) v1 ==== 221+0+0 (secure 0 0 0) 0x7f831805a400 con 0x7f832010bbd0 2024-03-26T02:35:52.135 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:52.133+0000 7f83260bb640 1 -- 172.21.15.72:0/626224747 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "100"} v 0) v1 -- 0x7f83201005a0 con 0x7f832010bbd0 2024-03-26T02:35:52.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:35:52 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:35:53.031 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:53.029+0000 7f831cff9640 1 -- 172.21.15.72:0/626224747 <== mon.2 v2:172.21.15.72:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool de5f38de-babb-481b-9bb5-ced33138378b v827) v1 ==== 221+0+0 (secure 0 0 0) 0x7f831805a5e0 con 0x7f832010bbd0 2024-03-26T02:35:53.031 INFO:tasks.workunit.client.0.smithi072.stderr:set-quota max_bytes = 100 for pool de5f38de-babb-481b-9bb5-ced33138378b 2024-03-26T02:35:53.033 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:53.031+0000 7f83260bb640 1 -- 172.21.15.72:0/626224747 >> v2:172.21.15.72:6800/3246690770 conn(0x7f830806d9f0 msgr2=0x7f830806feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:35:53.034 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:53.031+0000 7f83260bb640 1 --2- 172.21.15.72:0/626224747 >> v2:172.21.15.72:6800/3246690770 conn(0x7f830806d9f0 0x7f830806feb0 secure :-1 s=READY pgs=3384 cs=0 l=1 rev1=1 crypto rx=0x7f8310005b60 tx=0x7f8310000bc0 comp rx=0 tx=0).stop 2024-03-26T02:35:53.034 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:53.031+0000 7f83260bb640 1 -- 172.21.15.72:0/626224747 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f832010bbd0 msgr2=0x7f832019bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:35:53.034 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:53.031+0000 7f83260bb640 1 --2- 172.21.15.72:0/626224747 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f832010bbd0 0x7f832019bbd0 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7f831800c3f0 tx=0x7f831800c8b0 comp rx=0 tx=0).stop 2024-03-26T02:35:53.034 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:53.031+0000 7f83260bb640 1 -- 172.21.15.72:0/626224747 shutdown_connections 2024-03-26T02:35:53.034 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:53.031+0000 7f83260bb640 1 --2- 172.21.15.72:0/626224747 >> v2:172.21.15.72:6800/3246690770 conn(0x7f830806d9f0 0x7f830806feb0 unknown :-1 s=CLOSED pgs=3384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:53.034 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:53.031+0000 7f83260bb640 1 --2- 172.21.15.72:0/626224747 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f832010bbd0 0x7f832019bbd0 unknown :-1 s=CLOSED pgs=2562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:53.034 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:53.031+0000 7f83260bb640 1 --2- 172.21.15.72:0/626224747 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f8320103120 0x7f8320197850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:53.034 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:53.031+0000 7f83260bb640 1 --2- 172.21.15.72:0/626224747 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8320102800 0x7f8320197310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:35:53.035 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:53.031+0000 7f83260bb640 1 -- 172.21.15.72:0/626224747 >> 172.21.15.72:0/626224747 conn(0x7f8320077580 msgr2=0x7f83200fd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:35:53.035 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:53.032+0000 7f83260bb640 1 -- 172.21.15.72:0/626224747 shutdown_connections 2024-03-26T02:35:53.035 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:35:53.032+0000 7f83260bb640 1 -- 172.21.15.72:0/626224747 wait complete. 2024-03-26T02:35:53.044 INFO:tasks.workunit.client.0.smithi072.stderr:+ sleep 30 2024-03-26T02:35:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:53 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T02:35:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:53 smithi173 ceph-mon[33839]: osdmap e826: 8 total, 8 up, 8 in 2024-03-26T02:35:53.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:53 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/626224747' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T02:35:53.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:53 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T02:35:53.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:53 smithi173 ceph-mon[33839]: pgmap v1779: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 877 B/s rd, 0 op/s 2024-03-26T02:35:53.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:53 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:53.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:53 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T02:35:53.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:53 smithi072 ceph-mon[41619]: osdmap e826: 8 total, 8 up, 8 in 2024-03-26T02:35:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:53 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/626224747' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T02:35:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:53 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T02:35:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:53 smithi072 ceph-mon[41619]: pgmap v1779: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 877 B/s rd, 0 op/s 2024-03-26T02:35:53.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:53 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:53 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T02:35:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:53 smithi072 ceph-mon[30177]: osdmap e826: 8 total, 8 up, 8 in 2024-03-26T02:35:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:53 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/626224747' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T02:35:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:53 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T02:35:53.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:53 smithi072 ceph-mon[30177]: pgmap v1779: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 877 B/s rd, 0 op/s 2024-03-26T02:35:53.471 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:53 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:35:54.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:54 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T02:35:54.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:54 smithi173 ceph-mon[33839]: osdmap e827: 8 total, 8 up, 8 in 2024-03-26T02:35:54.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:54 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:35:54.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:54 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:35:54.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:54 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T02:35:54.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:54 smithi072 ceph-mon[30177]: osdmap e827: 8 total, 8 up, 8 in 2024-03-26T02:35:54.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:54 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:35:54.470 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:54 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:35:54.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:54 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T02:35:54.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:54 smithi072 ceph-mon[41619]: osdmap e827: 8 total, 8 up, 8 in 2024-03-26T02:35:54.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:54 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:35:54.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:54 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:35:55.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:55 smithi173 ceph-mon[33839]: pgmap v1781: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:55.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:55 smithi072 ceph-mon[30177]: pgmap v1781: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:55.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:55 smithi072 ceph-mon[41619]: pgmap v1781: 176 pgs: 176 active+clean; 698 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:35:55.470 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:35:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:35:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:35:56.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:56 smithi072 ceph-mon[30177]: pool 'de5f38de-babb-481b-9bb5-ced33138378b' is full (reached quota's max_bytes: 100 B) 2024-03-26T02:35:56.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:56 smithi072 ceph-mon[30177]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T02:35:56.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:56 smithi072 ceph-mon[30177]: osdmap e828: 8 total, 8 up, 8 in 2024-03-26T02:35:56.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:56 smithi072 ceph-mon[41619]: pool 'de5f38de-babb-481b-9bb5-ced33138378b' is full (reached quota's max_bytes: 100 B) 2024-03-26T02:35:56.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:56 smithi072 ceph-mon[41619]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T02:35:56.220 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:56 smithi072 ceph-mon[41619]: osdmap e828: 8 total, 8 up, 8 in 2024-03-26T02:35:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:56 smithi173 ceph-mon[33839]: pool 'de5f38de-babb-481b-9bb5-ced33138378b' is full (reached quota's max_bytes: 100 B) 2024-03-26T02:35:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:56 smithi173 ceph-mon[33839]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T02:35:56.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:56 smithi173 ceph-mon[33839]: osdmap e828: 8 total, 8 up, 8 in 2024-03-26T02:35:57.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:57 smithi173 ceph-mon[33839]: pgmap v1783: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 991 B/s rd, 4.6 KiB/s wr, 1 op/s 2024-03-26T02:35:57.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:57 smithi072 ceph-mon[30177]: pgmap v1783: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 991 B/s rd, 4.6 KiB/s wr, 1 op/s 2024-03-26T02:35:57.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:57 smithi072 ceph-mon[41619]: pgmap v1783: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 991 B/s rd, 4.6 KiB/s wr, 1 op/s 2024-03-26T02:35:59.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:35:59 smithi173 ceph-mon[33839]: pgmap v1784: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 831 B/s rd, 3.9 KiB/s wr, 1 op/s 2024-03-26T02:35:59.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:35:59 smithi072 ceph-mon[30177]: pgmap v1784: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 831 B/s rd, 3.9 KiB/s wr, 1 op/s 2024-03-26T02:35:59.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:35:59 smithi072 ceph-mon[41619]: pgmap v1784: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 831 B/s rd, 3.9 KiB/s wr, 1 op/s 2024-03-26T02:36:01.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:01 smithi173 ceph-mon[33839]: pgmap v1785: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 3.0 KiB/s wr, 0 op/s 2024-03-26T02:36:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:01 smithi072 ceph-mon[30177]: pgmap v1785: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 3.0 KiB/s wr, 0 op/s 2024-03-26T02:36:01.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:01 smithi072 ceph-mon[41619]: pgmap v1785: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 3.0 KiB/s wr, 0 op/s 2024-03-26T02:36:02.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:36:02 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:36:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:03 smithi173 ceph-mon[33839]: pgmap v1786: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.6 KiB/s wr, 1 op/s 2024-03-26T02:36:03.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:03 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:03.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:03 smithi072 ceph-mon[41619]: pgmap v1786: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.6 KiB/s wr, 1 op/s 2024-03-26T02:36:03.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:03 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:03.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:03 smithi072 ceph-mon[30177]: pgmap v1786: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 2.6 KiB/s wr, 1 op/s 2024-03-26T02:36:03.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:03 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:05.245 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:36:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:36:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:36:05.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:05 smithi173 ceph-mon[33839]: pgmap v1787: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.4 KiB/s wr, 1 op/s 2024-03-26T02:36:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:05 smithi072 ceph-mon[30177]: pgmap v1787: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.4 KiB/s wr, 1 op/s 2024-03-26T02:36:05.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:05 smithi072 ceph-mon[41619]: pgmap v1787: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.4 KiB/s wr, 1 op/s 2024-03-26T02:36:07.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:07 smithi173 ceph-mon[33839]: pgmap v1788: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2024-03-26T02:36:07.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:07 smithi072 ceph-mon[30177]: pgmap v1788: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2024-03-26T02:36:07.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:07 smithi072 ceph-mon[41619]: pgmap v1788: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1007 B/s rd, 0 op/s 2024-03-26T02:36:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:09 smithi072 ceph-mon[30177]: pgmap v1789: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:09 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:36:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:09 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:36:09.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:09 smithi072 ceph-mon[41619]: pgmap v1789: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:09.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:09 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:36:09.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:09 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:36:09.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:09 smithi173 ceph-mon[33839]: pgmap v1789: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:09.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:09 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:36:09.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:09 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:36:11.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:11 smithi072 ceph-mon[30177]: pgmap v1790: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:11.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:11 smithi072 ceph-mon[41619]: pgmap v1790: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:11.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:11 smithi173 ceph-mon[33839]: pgmap v1790: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:12.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:36:12 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:36:13.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:13 smithi072 ceph-mon[30177]: pgmap v1791: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:13.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:13 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:13.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:13 smithi072 ceph-mon[41619]: pgmap v1791: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:13.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:13 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:13.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:13 smithi173 ceph-mon[33839]: pgmap v1791: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:13.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:13 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:15.368 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:36:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:36:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:36:15.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:15 smithi072 ceph-mon[30177]: pgmap v1792: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:15.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:15 smithi072 ceph-mon[41619]: pgmap v1792: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:15.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:15 smithi173 ceph-mon[33839]: pgmap v1792: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:17.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:17 smithi072 ceph-mon[30177]: pgmap v1793: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:17.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:17 smithi072 ceph-mon[41619]: pgmap v1793: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:17.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:17 smithi173 ceph-mon[33839]: pgmap v1793: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:19.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:19 smithi072 ceph-mon[30177]: pgmap v1794: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:19.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:19 smithi072 ceph-mon[41619]: pgmap v1794: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:19 smithi173 ceph-mon[33839]: pgmap v1794: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:21.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:21 smithi072 ceph-mon[30177]: pgmap v1795: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:21.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:21 smithi072 ceph-mon[41619]: pgmap v1795: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:21.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:21 smithi173 ceph-mon[33839]: pgmap v1795: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:22.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:36:22 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:36:23.046 INFO:tasks.workunit.client.0.smithi072.stderr:+ pid=159413 2024-03-26T02:36:23.046 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph osd pool set-quota de5f38de-babb-481b-9bb5-ced33138378b max_bytes 0 2024-03-26T02:36:23.046 INFO:tasks.workunit.client.0.smithi072.stderr:+ rados -p de5f38de-babb-481b-9bb5-ced33138378b put two /etc/passwd 2024-03-26T02:36:23.143 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.139+0000 7f33ffc15640 1 -- 172.21.15.72:0/271398514 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f33f8101580 msgr2=0x7f33f8101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:36:23.143 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.139+0000 7f33ffc15640 1 --2- 172.21.15.72:0/271398514 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f33f8101580 0x7f33f8101a00 secure :-1 s=READY pgs=2563 cs=0 l=1 rev1=1 crypto rx=0x7f33e8009890 tx=0x7f33e802bb70 comp rx=0 tx=0).stop 2024-03-26T02:36:23.143 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.139+0000 7f33ffc15640 1 -- 172.21.15.72:0/271398514 shutdown_connections 2024-03-26T02:36:23.143 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.139+0000 7f33ffc15640 1 --2- 172.21.15.72:0/271398514 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33f8101f40 0x7f33f810e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:23.143 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.139+0000 7f33ffc15640 1 --2- 172.21.15.72:0/271398514 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f33f8101580 0x7f33f8101a00 unknown :-1 s=CLOSED pgs=2563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:23.143 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.139+0000 7f33ffc15640 1 --2- 172.21.15.72:0/271398514 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f33f8107640 0x7f33f8107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:23.143 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.139+0000 7f33ffc15640 1 -- 172.21.15.72:0/271398514 >> 172.21.15.72:0/271398514 conn(0x7f33f80fd070 msgr2=0x7f33f80ff490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:36:23.143 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.139+0000 7f33ffc15640 1 -- 172.21.15.72:0/271398514 shutdown_connections 2024-03-26T02:36:23.143 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.139+0000 7f33ffc15640 1 -- 172.21.15.72:0/271398514 wait complete. 2024-03-26T02:36:23.143 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.140+0000 7f33ffc15640 1 Processor -- start 2024-03-26T02:36:23.144 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.140+0000 7f33ffc15640 1 -- start start 2024-03-26T02:36:23.144 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.140+0000 7f33ffc15640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33f8101580 0x7f33f8197310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:36:23.144 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.140+0000 7f33ffc15640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f33f8101f40 0x7f33f8197850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:36:23.144 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.140+0000 7f33ffc15640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f33f8107640 0x7f33f819bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:36:23.144 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.140+0000 7f33ffc15640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f819c1d0 con 0x7f33f8107640 2024-03-26T02:36:23.144 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.140+0000 7f33ffc15640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f819c340 con 0x7f33f8101580 2024-03-26T02:36:23.144 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.140+0000 7f33ffc15640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33f819c500 con 0x7f33f8101f40 2024-03-26T02:36:23.144 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.140+0000 7f33fd1b0640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f33f8101f40 0x7f33f8197850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:36:23.144 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.140+0000 7f33fe1b2640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f33f8107640 0x7f33f819bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:36:23.144 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.140+0000 7f33fd1b0640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f33f8101f40 0x7f33f8197850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:35880/0 (socket says 172.21.15.72:35880) 2024-03-26T02:36:23.144 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.140+0000 7f33fd1b0640 1 -- 172.21.15.72:0/221217865 learned_addr learned my addr 172.21.15.72:0/221217865 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:36:23.144 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.140+0000 7f33fd9b1640 1 --2- 172.21.15.72:0/221217865 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33f8101580 0x7f33f8197310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:36:23.144 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.141+0000 7f33fe1b2640 1 -- 172.21.15.72:0/221217865 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f33f8101f40 msgr2=0x7f33f8197850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:36:23.144 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.141+0000 7f33fe1b2640 1 --2- 172.21.15.72:0/221217865 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f33f8101f40 0x7f33f8197850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:23.144 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.141+0000 7f33fe1b2640 1 -- 172.21.15.72:0/221217865 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33f8101580 msgr2=0x7f33f8197310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:36:23.144 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.141+0000 7f33fe1b2640 1 --2- 172.21.15.72:0/221217865 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33f8101580 0x7f33f8197310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:23.144 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.141+0000 7f33fe1b2640 1 -- 172.21.15.72:0/221217865 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f4009560 con 0x7f33f8107640 2024-03-26T02:36:23.145 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.141+0000 7f33fd1b0640 1 --2- 172.21.15.72:0/221217865 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f33f8101f40 0x7f33f8197850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:36:23.145 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.141+0000 7f33fe1b2640 1 --2- 172.21.15.72:0/221217865 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f33f8107640 0x7f33f819bbd0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f33f000ca30 tx=0x7f33f000cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:36:23.145 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.141+0000 7f33eeffd640 1 -- 172.21.15.72:0/221217865 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f0007d90 con 0x7f33f8107640 2024-03-26T02:36:23.145 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.141+0000 7f33eeffd640 1 -- 172.21.15.72:0/221217865 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f33f0005020 con 0x7f33f8107640 2024-03-26T02:36:23.145 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.141+0000 7f33ffc15640 1 -- 172.21.15.72:0/221217865 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33e8009490 con 0x7f33f8107640 2024-03-26T02:36:23.145 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.141+0000 7f33eeffd640 1 -- 172.21.15.72:0/221217865 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f000f690 con 0x7f33f8107640 2024-03-26T02:36:23.145 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.141+0000 7f33ffc15640 1 -- 172.21.15.72:0/221217865 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f81a3e80 con 0x7f33f8107640 2024-03-26T02:36:23.145 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.142+0000 7f33ffc15640 1 -- 172.21.15.72:0/221217865 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33f80688c0 con 0x7f33f8107640 2024-03-26T02:36:23.146 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.142+0000 7f33eeffd640 1 -- 172.21.15.72:0/221217865 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7f33f0004a40 con 0x7f33f8107640 2024-03-26T02:36:23.146 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.143+0000 7f33eeffd640 1 --2- 172.21.15.72:0/221217865 >> v2:172.21.15.72:6800/3246690770 conn(0x7f33e406d9f0 0x7f33e406feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:36:23.146 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.143+0000 7f33fd9b1640 1 --2- 172.21.15.72:0/221217865 >> v2:172.21.15.72:6800/3246690770 conn(0x7f33e406d9f0 0x7f33e406feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:36:23.146 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.143+0000 7f33eeffd640 1 -- 172.21.15.72:0/221217865 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(828..828 src has 255..828) v4 ==== 7846+0+0 (secure 0 0 0) 0x7f33f008dc90 con 0x7f33f8107640 2024-03-26T02:36:23.146 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.143+0000 7f33eeffd640 1 -- 172.21.15.72:0/221217865 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=829}) v3 -- 0x7f33e4078f60 con 0x7f33f8107640 2024-03-26T02:36:23.147 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.143+0000 7f33fd9b1640 1 --2- 172.21.15.72:0/221217865 >> v2:172.21.15.72:6800/3246690770 conn(0x7f33e406d9f0 0x7f33e406feb0 secure :-1 s=READY pgs=3386 cs=0 l=1 rev1=1 crypto rx=0x7f33f4009530 tx=0x7f33f4009260 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:36:23.148 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.148+0000 7f33eeffd640 1 -- 172.21.15.72:0/221217865 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f33f001d060 con 0x7f33f8107640 2024-03-26T02:36:23.324 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.323+0000 7f33ffc15640 1 -- 172.21.15.72:0/221217865 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f33f8110d80 con 0x7f33f8107640 2024-03-26T02:36:23.386 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.385+0000 7f33eeffd640 1 -- 172.21.15.72:0/221217865 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool de5f38de-babb-481b-9bb5-ced33138378b v829) v1 ==== 217+0+0 (secure 0 0 0) 0x7f33f005ab30 con 0x7f33f8107640 2024-03-26T02:36:23.387 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.386+0000 7f33eeffd640 1 -- 172.21.15.72:0/221217865 <== mon.0 v2:172.21.15.72:3300/0 8 ==== osd_map(829..829 src has 255..829) v4 ==== 628+0+0 (secure 0 0 0) 0x7f33f0005490 con 0x7f33f8107640 2024-03-26T02:36:23.387 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.386+0000 7f33eeffd640 1 -- 172.21.15.72:0/221217865 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=830}) v3 -- 0x7f33e4079e10 con 0x7f33f8107640 2024-03-26T02:36:23.494 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:23.493+0000 7f33ffc15640 1 -- 172.21.15.72:0/221217865 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f33f8110f90 con 0x7f33f8107640 2024-03-26T02:36:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:23 smithi072 ceph-mon[30177]: pgmap v1796: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:23 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:23 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/221217865' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T02:36:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:23 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:36:23.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:23 smithi072 ceph-mon[41619]: pgmap v1796: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:23.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:23 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:23 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/221217865' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T02:36:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:23 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:36:23.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:23 smithi173 ceph-mon[33839]: pgmap v1796: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:23.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:23 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:23.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:23 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/221217865' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T02:36:23.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:23 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:36:24.392 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.391+0000 7f33eeffd640 1 -- 172.21.15.72:0/221217865 <== mon.0 v2:172.21.15.72:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool de5f38de-babb-481b-9bb5-ced33138378b v830) v1 ==== 217+0+0 (secure 0 0 0) 0x7f33f005a6c0 con 0x7f33f8107640 2024-03-26T02:36:24.392 INFO:tasks.workunit.client.0.smithi072.stderr:set-quota max_bytes = 0 for pool de5f38de-babb-481b-9bb5-ced33138378b 2024-03-26T02:36:24.393 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.392+0000 7f33eeffd640 1 -- 172.21.15.72:0/221217865 <== mon.0 v2:172.21.15.72:3300/0 10 ==== osd_map(830..830 src has 255..830) v4 ==== 628+0+0 (secure 0 0 0) 0x7f33f00610d0 con 0x7f33f8107640 2024-03-26T02:36:24.393 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.392+0000 7f33eeffd640 1 -- 172.21.15.72:0/221217865 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=831}) v3 -- 0x7f33e4079fe0 con 0x7f33f8107640 2024-03-26T02:36:24.396 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.395+0000 7f33ffc15640 1 -- 172.21.15.72:0/221217865 >> v2:172.21.15.72:6800/3246690770 conn(0x7f33e406d9f0 msgr2=0x7f33e406feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:36:24.396 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.395+0000 7f33ffc15640 1 --2- 172.21.15.72:0/221217865 >> v2:172.21.15.72:6800/3246690770 conn(0x7f33e406d9f0 0x7f33e406feb0 secure :-1 s=READY pgs=3386 cs=0 l=1 rev1=1 crypto rx=0x7f33f4009530 tx=0x7f33f4009260 comp rx=0 tx=0).stop 2024-03-26T02:36:24.396 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.395+0000 7f33ffc15640 1 -- 172.21.15.72:0/221217865 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f33f8107640 msgr2=0x7f33f819bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:36:24.396 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.395+0000 7f33ffc15640 1 --2- 172.21.15.72:0/221217865 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f33f8107640 0x7f33f819bbd0 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f33f000ca30 tx=0x7f33f000cef0 comp rx=0 tx=0).stop 2024-03-26T02:36:24.396 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.395+0000 7f33ffc15640 1 -- 172.21.15.72:0/221217865 shutdown_connections 2024-03-26T02:36:24.396 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.395+0000 7f33ffc15640 1 --2- 172.21.15.72:0/221217865 >> v2:172.21.15.72:6800/3246690770 conn(0x7f33e406d9f0 0x7f33e406feb0 unknown :-1 s=CLOSED pgs=3386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:24.396 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.395+0000 7f33ffc15640 1 --2- 172.21.15.72:0/221217865 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f33f8107640 0x7f33f819bbd0 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:24.397 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.395+0000 7f33ffc15640 1 --2- 172.21.15.72:0/221217865 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f33f8101f40 0x7f33f8197850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:24.397 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.395+0000 7f33ffc15640 1 --2- 172.21.15.72:0/221217865 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33f8101580 0x7f33f8197310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:24.397 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.395+0000 7f33ffc15640 1 -- 172.21.15.72:0/221217865 >> 172.21.15.72:0/221217865 conn(0x7f33f80fd070 msgr2=0x7f33f8105690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:36:24.397 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.395+0000 7f33ffc15640 1 -- 172.21.15.72:0/221217865 shutdown_connections 2024-03-26T02:36:24.397 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.395+0000 7f33ffc15640 1 -- 172.21.15.72:0/221217865 wait complete. 2024-03-26T02:36:24.409 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph osd pool set-quota de5f38de-babb-481b-9bb5-ced33138378b max_objects 0 2024-03-26T02:36:24.498 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.495+0000 7fe96c258640 1 -- 172.21.15.72:0/1443742407 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe964102800 msgr2=0x7fe964102be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:36:24.498 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.495+0000 7fe96c258640 1 --2- 172.21.15.72:0/1443742407 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe964102800 0x7fe964102be0 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7fe960009860 tx=0x7fe96002bca0 comp rx=0 tx=0).stop 2024-03-26T02:36:24.498 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.495+0000 7fe96c258640 1 -- 172.21.15.72:0/1443742407 shutdown_connections 2024-03-26T02:36:24.499 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.495+0000 7fe96c258640 1 --2- 172.21.15.72:0/1443742407 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe96410bb40 0x7fe96410dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:24.499 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.495+0000 7fe96c258640 1 --2- 172.21.15.72:0/1443742407 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe964103120 0x7fe96410b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:24.499 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.495+0000 7fe96c258640 1 --2- 172.21.15.72:0/1443742407 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe964102800 0x7fe964102be0 unknown :-1 s=CLOSED pgs=2564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:24.499 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.495+0000 7fe96c258640 1 -- 172.21.15.72:0/1443742407 >> 172.21.15.72:0/1443742407 conn(0x7fe9640775b0 msgr2=0x7fe9640779c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:36:24.499 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.495+0000 7fe96c258640 1 -- 172.21.15.72:0/1443742407 shutdown_connections 2024-03-26T02:36:24.499 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.495+0000 7fe96c258640 1 -- 172.21.15.72:0/1443742407 wait complete. 2024-03-26T02:36:24.499 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.495+0000 7fe96c258640 1 Processor -- start 2024-03-26T02:36:24.499 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.495+0000 7fe96c258640 1 -- start start 2024-03-26T02:36:24.499 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.496+0000 7fe96c258640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe964102800 0x7fe964197340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:36:24.499 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.496+0000 7fe96c258640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe964103120 0x7fe964197880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:36:24.499 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.496+0000 7fe96c258640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe96410bb40 0x7fe96419bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:36:24.499 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.496+0000 7fe96c258640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96419c1b0 con 0x7fe96410bb40 2024-03-26T02:36:24.499 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.496+0000 7fe96c258640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96419c320 con 0x7fe964103120 2024-03-26T02:36:24.500 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.496+0000 7fe96c258640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe96419c4e0 con 0x7fe964102800 2024-03-26T02:36:24.500 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.496+0000 7fe969ff4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe964102800 0x7fe964197340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:36:24.500 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.496+0000 7fe969ff4640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe964102800 0x7fe964197340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:35906/0 (socket says 172.21.15.72:35906) 2024-03-26T02:36:24.500 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.496+0000 7fe969ff4640 1 -- 172.21.15.72:0/3505804172 learned_addr learned my addr 172.21.15.72:0/3505804172 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:36:24.500 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.496+0000 7fe96a7f5640 1 --2- 172.21.15.72:0/3505804172 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe96410bb40 0x7fe96419bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:36:24.500 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.496+0000 7fe9697f3640 1 --2- 172.21.15.72:0/3505804172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe964103120 0x7fe964197880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:36:24.500 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.496+0000 7fe969ff4640 1 -- 172.21.15.72:0/3505804172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe964103120 msgr2=0x7fe964197880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:36:24.500 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.496+0000 7fe969ff4640 1 --2- 172.21.15.72:0/3505804172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe964103120 0x7fe964197880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:24.500 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.496+0000 7fe969ff4640 1 -- 172.21.15.72:0/3505804172 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe96410bb40 msgr2=0x7fe96419bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:36:24.500 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.496+0000 7fe969ff4640 1 --2- 172.21.15.72:0/3505804172 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe96410bb40 0x7fe96419bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:24.500 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.496+0000 7fe969ff4640 1 -- 172.21.15.72:0/3505804172 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95c009900 con 0x7fe964102800 2024-03-26T02:36:24.500 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.496+0000 7fe969ff4640 1 --2- 172.21.15.72:0/3505804172 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe964102800 0x7fe964197340 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7fe960037040 tx=0x7fe96002ca30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:36:24.501 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.496+0000 7fe95affd640 1 -- 172.21.15.72:0/3505804172 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe960034640 con 0x7fe964102800 2024-03-26T02:36:24.501 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.496+0000 7fe95affd640 1 -- 172.21.15.72:0/3505804172 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fe9600035f0 con 0x7fe964102800 2024-03-26T02:36:24.501 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.496+0000 7fe96c258640 1 -- 172.21.15.72:0/3505804172 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe960009490 con 0x7fe964102800 2024-03-26T02:36:24.501 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.496+0000 7fe95affd640 1 -- 172.21.15.72:0/3505804172 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe96003d630 con 0x7fe964102800 2024-03-26T02:36:24.501 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.496+0000 7fe96c258640 1 -- 172.21.15.72:0/3505804172 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9641a3e60 con 0x7fe964102800 2024-03-26T02:36:24.501 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.497+0000 7fe96c258640 1 -- 172.21.15.72:0/3505804172 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe964068470 con 0x7fe964102800 2024-03-26T02:36:24.502 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.498+0000 7fe95affd640 1 -- 172.21.15.72:0/3505804172 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7fe96003b040 con 0x7fe964102800 2024-03-26T02:36:24.502 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.498+0000 7fe95affd640 1 --2- 172.21.15.72:0/3505804172 >> v2:172.21.15.72:6800/3246690770 conn(0x7fe95006d9f0 0x7fe95006feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:36:24.502 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.498+0000 7fe95affd640 1 -- 172.21.15.72:0/3505804172 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(830..830 src has 255..830) v4 ==== 7846+0+0 (secure 0 0 0) 0x7fe9600af530 con 0x7fe964102800 2024-03-26T02:36:24.502 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.498+0000 7fe9697f3640 1 --2- 172.21.15.72:0/3505804172 >> v2:172.21.15.72:6800/3246690770 conn(0x7fe95006d9f0 0x7fe95006feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:36:24.502 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.499+0000 7fe95affd640 1 -- 172.21.15.72:0/3505804172 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=831}) v3 -- 0x7fe950078f60 con 0x7fe964102800 2024-03-26T02:36:24.502 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.499+0000 7fe9697f3640 1 --2- 172.21.15.72:0/3505804172 >> v2:172.21.15.72:6800/3246690770 conn(0x7fe95006d9f0 0x7fe95006feb0 secure :-1 s=READY pgs=3387 cs=0 l=1 rev1=1 crypto rx=0x7fe954005fd0 tx=0x7fe954009470 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:36:24.504 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.504+0000 7fe95affd640 1 -- 172.21.15.72:0/3505804172 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fe960078020 con 0x7fe964102800 2024-03-26T02:36:24.704 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:24.703+0000 7fe96c258640 1 -- 172.21.15.72:0/3505804172 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7fe9641063d0 con 0x7fe964102800 2024-03-26T02:36:24.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:24 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/221217865' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T02:36:24.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:24 smithi072 ceph-mon[30177]: osdmap e829: 8 total, 8 up, 8 in 2024-03-26T02:36:24.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:24 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/221217865' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T02:36:24.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:24 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/221217865' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T02:36:24.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:24 smithi072 ceph-mon[41619]: osdmap e829: 8 total, 8 up, 8 in 2024-03-26T02:36:24.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:24 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/221217865' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T02:36:24.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:24 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/221217865' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T02:36:24.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:24 smithi173 ceph-mon[33839]: osdmap e829: 8 total, 8 up, 8 in 2024-03-26T02:36:24.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:24 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/221217865' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T02:36:25.394 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:36:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:36:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:36:25.401 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:25.400+0000 7fe95affd640 1 -- 172.21.15.72:0/3505804172 <== mon.2 v2:172.21.15.72:3301/0 7 ==== osd_map(831..831 src has 255..831) v4 ==== 628+0+0 (secure 0 0 0) 0x7fe96009c0c0 con 0x7fe964102800 2024-03-26T02:36:25.401 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:25.400+0000 7fe95affd640 1 -- 172.21.15.72:0/3505804172 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=832}) v3 -- 0x7fe950079a90 con 0x7fe964102800 2024-03-26T02:36:25.405 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:25.404+0000 7fe95affd640 1 -- 172.21.15.72:0/3505804172 <== mon.2 v2:172.21.15.72:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool de5f38de-babb-481b-9bb5-ced33138378b v831) v1 ==== 221+0+0 (secure 0 0 0) 0x7fe960039030 con 0x7fe964102800 2024-03-26T02:36:25.511 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:25.510+0000 7fe96c258640 1 -- 172.21.15.72:0/3505804172 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7fe9641063d0 con 0x7fe964102800 2024-03-26T02:36:25.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:25 smithi072 ceph-mon[30177]: pgmap v1798: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:36:25.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/221217865' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T02:36:25.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:25 smithi072 ceph-mon[30177]: osdmap e830: 8 total, 8 up, 8 in 2024-03-26T02:36:25.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:25 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:36:25.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:25 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3505804172' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T02:36:25.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:25 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T02:36:25.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:25 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:36:25.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:25 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:36:25.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:25 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:36:25.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:25 smithi072 ceph-mon[41619]: pgmap v1798: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:36:25.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/221217865' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T02:36:25.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:25 smithi072 ceph-mon[41619]: osdmap e830: 8 total, 8 up, 8 in 2024-03-26T02:36:25.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:25 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:36:25.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:25 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3505804172' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T02:36:25.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:25 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T02:36:25.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:25 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:36:25.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:25 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:36:25.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:25 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:36:25.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:25 smithi173 ceph-mon[33839]: pgmap v1798: 176 pgs: 176 active+clean; 721 KiB data, 894 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:36:25.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/221217865' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T02:36:25.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:25 smithi173 ceph-mon[33839]: osdmap e830: 8 total, 8 up, 8 in 2024-03-26T02:36:25.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:25 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:36:25.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:25 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3505804172' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T02:36:25.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:25 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T02:36:25.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:25 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:36:25.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:25 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:36:25.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:25 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:36:26.025 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.024+0000 7fe95affd640 1 -- 172.21.15.72:0/3505804172 <== mon.2 v2:172.21.15.72:3301/0 9 ==== osd_map(832..832 src has 255..832) v4 ==== 628+0+0 (secure 0 0 0) 0x7fe9600806a0 con 0x7fe964102800 2024-03-26T02:36:26.025 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.024+0000 7fe95affd640 1 -- 172.21.15.72:0/3505804172 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=833}) v3 -- 0x7fe950079cc0 con 0x7fe964102800 2024-03-26T02:36:26.030 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.029+0000 7fe95affd640 1 -- 172.21.15.72:0/3505804172 <== mon.2 v2:172.21.15.72:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool de5f38de-babb-481b-9bb5-ced33138378b v832) v1 ==== 221+0+0 (secure 0 0 0) 0x7fe960039030 con 0x7fe964102800 2024-03-26T02:36:26.030 INFO:tasks.workunit.client.0.smithi072.stderr:set-quota max_objects = 0 for pool de5f38de-babb-481b-9bb5-ced33138378b 2024-03-26T02:36:26.033 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.032+0000 7fe96c258640 1 -- 172.21.15.72:0/3505804172 >> v2:172.21.15.72:6800/3246690770 conn(0x7fe95006d9f0 msgr2=0x7fe95006feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:36:26.033 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.032+0000 7fe96c258640 1 --2- 172.21.15.72:0/3505804172 >> v2:172.21.15.72:6800/3246690770 conn(0x7fe95006d9f0 0x7fe95006feb0 secure :-1 s=READY pgs=3387 cs=0 l=1 rev1=1 crypto rx=0x7fe954005fd0 tx=0x7fe954009470 comp rx=0 tx=0).stop 2024-03-26T02:36:26.034 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.032+0000 7fe96c258640 1 -- 172.21.15.72:0/3505804172 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe964102800 msgr2=0x7fe964197340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:36:26.034 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.032+0000 7fe96c258640 1 --2- 172.21.15.72:0/3505804172 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe964102800 0x7fe964197340 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7fe960037040 tx=0x7fe96002ca30 comp rx=0 tx=0).stop 2024-03-26T02:36:26.034 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.032+0000 7fe96c258640 1 -- 172.21.15.72:0/3505804172 shutdown_connections 2024-03-26T02:36:26.034 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.032+0000 7fe96c258640 1 --2- 172.21.15.72:0/3505804172 >> v2:172.21.15.72:6800/3246690770 conn(0x7fe95006d9f0 0x7fe95006feb0 unknown :-1 s=CLOSED pgs=3387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:26.034 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.032+0000 7fe96c258640 1 --2- 172.21.15.72:0/3505804172 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fe96410bb40 0x7fe96419bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:26.034 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.032+0000 7fe96c258640 1 --2- 172.21.15.72:0/3505804172 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe964103120 0x7fe964197880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:26.034 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.032+0000 7fe96c258640 1 --2- 172.21.15.72:0/3505804172 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fe964102800 0x7fe964197340 unknown :-1 s=CLOSED pgs=2565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:26.034 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.032+0000 7fe96c258640 1 -- 172.21.15.72:0/3505804172 >> 172.21.15.72:0/3505804172 conn(0x7fe9640775b0 msgr2=0x7fe9641035a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:36:26.034 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.032+0000 7fe96c258640 1 -- 172.21.15.72:0/3505804172 shutdown_connections 2024-03-26T02:36:26.034 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.032+0000 7fe96c258640 1 -- 172.21.15.72:0/3505804172 wait complete. 2024-03-26T02:36:26.043 INFO:tasks.workunit.client.0.smithi072.stderr:+ wait 159413 2024-03-26T02:36:26.043 INFO:tasks.workunit.client.0.smithi072.stderr:+ '[' 0 -ne 0 ']' 2024-03-26T02:36:26.043 INFO:tasks.workunit.client.0.smithi072.stderr:+ true 2024-03-26T02:36:26.043 INFO:tasks.workunit.client.0.smithi072.stderr:+ rados -p de5f38de-babb-481b-9bb5-ced33138378b put three /etc/passwd 2024-03-26T02:36:26.080 INFO:tasks.workunit.client.0.smithi072.stderr:++ uuidgen 2024-03-26T02:36:26.080 INFO:tasks.workunit.client.0.smithi072.stderr:+ pp=6857fade-a9dd-4b5e-87a3-010f68d73cf2 2024-03-26T02:36:26.080 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph osd pool create 6857fade-a9dd-4b5e-87a3-010f68d73cf2 12 2024-03-26T02:36:26.164 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.160+0000 7f143bd48640 1 -- 172.21.15.72:0/1346740025 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1434100e10 msgr2=0x7f143410d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:36:26.164 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.160+0000 7f143bd48640 1 --2- 172.21.15.72:0/1346740025 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1434100e10 0x7f143410d320 secure :-1 s=READY pgs=2568 cs=0 l=1 rev1=1 crypto rx=0x7f142c009cd0 tx=0x7f142c02c0f0 comp rx=0 tx=0).stop 2024-03-26T02:36:26.164 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.160+0000 7f143bd48640 1 -- 172.21.15.72:0/1346740025 shutdown_connections 2024-03-26T02:36:26.164 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.160+0000 7f143bd48640 1 --2- 172.21.15.72:0/1346740025 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1434100e10 0x7f143410d320 unknown :-1 s=CLOSED pgs=2568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:26.164 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.160+0000 7f143bd48640 1 --2- 172.21.15.72:0/1346740025 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1434100450 0x7f14341008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:26.164 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.161+0000 7f143bd48640 1 --2- 172.21.15.72:0/1346740025 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f14341064f0 0x7f14341068d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:26.164 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.161+0000 7f143bd48640 1 -- 172.21.15.72:0/1346740025 >> 172.21.15.72:0/1346740025 conn(0x7f14340775b0 msgr2=0x7f14340779c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:36:26.164 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.161+0000 7f143bd48640 1 -- 172.21.15.72:0/1346740025 shutdown_connections 2024-03-26T02:36:26.164 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.161+0000 7f143bd48640 1 -- 172.21.15.72:0/1346740025 wait complete. 2024-03-26T02:36:26.165 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.161+0000 7f143bd48640 1 Processor -- start 2024-03-26T02:36:26.165 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.161+0000 7f143bd48640 1 -- start start 2024-03-26T02:36:26.165 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.161+0000 7f143bd48640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1434100450 0x7f1434195f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:36:26.165 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.161+0000 7f143bd48640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1434100e10 0x7f1434196460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:36:26.165 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.161+0000 7f143bd48640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14341064f0 0x7f14341969a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:36:26.165 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.161+0000 7f143bd48640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14341a5630 con 0x7f1434100450 2024-03-26T02:36:26.165 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.161+0000 7f143bd48640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14341a57a0 con 0x7f14341064f0 2024-03-26T02:36:26.165 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.161+0000 7f143bd48640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14341a5960 con 0x7f1434100e10 2024-03-26T02:36:26.165 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.161+0000 7f1439ae4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1434100450 0x7f1434195f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:36:26.165 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.161+0000 7f14392e3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1434100e10 0x7f1434196460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:36:26.165 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.161+0000 7f1439ae4640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1434100450 0x7f1434195f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:35568/0 (socket says 172.21.15.72:35568) 2024-03-26T02:36:26.165 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.161+0000 7f14392e3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1434100e10 0x7f1434196460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:35934/0 (socket says 172.21.15.72:35934) 2024-03-26T02:36:26.165 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.161+0000 7f1439ae4640 1 -- 172.21.15.72:0/459206853 learned_addr learned my addr 172.21.15.72:0/459206853 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:36:26.165 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.162+0000 7f143a2e5640 1 --2- 172.21.15.72:0/459206853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14341064f0 0x7f14341969a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:36:26.166 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.162+0000 7f1439ae4640 1 -- 172.21.15.72:0/459206853 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1434100e10 msgr2=0x7f1434196460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:36:26.166 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.162+0000 7f1439ae4640 1 --2- 172.21.15.72:0/459206853 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1434100e10 0x7f1434196460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:26.166 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.162+0000 7f1439ae4640 1 -- 172.21.15.72:0/459206853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14341064f0 msgr2=0x7f14341969a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:36:26.166 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.162+0000 7f1439ae4640 1 --2- 172.21.15.72:0/459206853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14341064f0 0x7f14341969a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:26.166 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.162+0000 7f1439ae4640 1 -- 172.21.15.72:0/459206853 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1430009490 con 0x7f1434100450 2024-03-26T02:36:26.166 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.162+0000 7f143a2e5640 1 --2- 172.21.15.72:0/459206853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14341064f0 0x7f14341969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-26T02:36:26.166 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.162+0000 7f14392e3640 1 --2- 172.21.15.72:0/459206853 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1434100e10 0x7f1434196460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:36:26.166 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.162+0000 7f1439ae4640 1 --2- 172.21.15.72:0/459206853 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1434100450 0x7f1434195f20 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f143000ea60 tx=0x7f143000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:36:26.166 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.162+0000 7f142affd640 1 -- 172.21.15.72:0/459206853 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f143000cde0 con 0x7f1434100450 2024-03-26T02:36:26.166 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.162+0000 7f143bd48640 1 -- 172.21.15.72:0/459206853 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f142c009900 con 0x7f1434100450 2024-03-26T02:36:26.166 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.162+0000 7f142affd640 1 -- 172.21.15.72:0/459206853 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f14300044d0 con 0x7f1434100450 2024-03-26T02:36:26.166 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.162+0000 7f142affd640 1 -- 172.21.15.72:0/459206853 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1430010690 con 0x7f1434100450 2024-03-26T02:36:26.166 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.162+0000 7f143bd48640 1 -- 172.21.15.72:0/459206853 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14341a6010 con 0x7f1434100450 2024-03-26T02:36:26.167 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.163+0000 7f143bd48640 1 -- 172.21.15.72:0/459206853 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1434068470 con 0x7f1434100450 2024-03-26T02:36:26.167 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.164+0000 7f142affd640 1 -- 172.21.15.72:0/459206853 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7f1430003e10 con 0x7f1434100450 2024-03-26T02:36:26.167 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.164+0000 7f142affd640 1 --2- 172.21.15.72:0/459206853 >> v2:172.21.15.72:6800/3246690770 conn(0x7f142006d9f0 0x7f142006feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:36:26.168 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.164+0000 7f14392e3640 1 --2- 172.21.15.72:0/459206853 >> v2:172.21.15.72:6800/3246690770 conn(0x7f142006d9f0 0x7f142006feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:36:26.168 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.164+0000 7f142affd640 1 -- 172.21.15.72:0/459206853 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(832..832 src has 255..832) v4 ==== 7846+0+0 (secure 0 0 0) 0x7f1430014070 con 0x7f1434100450 2024-03-26T02:36:26.168 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.164+0000 7f14392e3640 1 --2- 172.21.15.72:0/459206853 >> v2:172.21.15.72:6800/3246690770 conn(0x7f142006d9f0 0x7f142006feb0 secure :-1 s=READY pgs=3389 cs=0 l=1 rev1=1 crypto rx=0x7f1434069db0 tx=0x7f142400b390 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:36:26.170 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.169+0000 7f142affd640 1 -- 172.21.15.72:0/459206853 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f143005ac90 con 0x7f1434100450 2024-03-26T02:36:26.344 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:26.343+0000 7f143bd48640 1 -- 172.21.15.72:0/459206853 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "pg_num": 12} v 0) v1 -- 0x7f143410fc50 con 0x7f1434100450 2024-03-26T02:36:26.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:26 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]': finished 2024-03-26T02:36:26.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:26 smithi072 ceph-mon[30177]: osdmap e831: 8 total, 8 up, 8 in 2024-03-26T02:36:26.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:26 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3505804172' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T02:36:26.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:26 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T02:36:26.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:26 smithi072 ceph-mon[30177]: pool 'de5f38de-babb-481b-9bb5-ced33138378b' no longer out of quota; removing NO_QUOTA flag 2024-03-26T02:36:26.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:26 smithi072 ceph-mon[30177]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T02:36:26.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:26 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]': finished 2024-03-26T02:36:26.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:26 smithi072 ceph-mon[30177]: osdmap e832: 8 total, 8 up, 8 in 2024-03-26T02:36:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:26 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]': finished 2024-03-26T02:36:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:26 smithi072 ceph-mon[41619]: osdmap e831: 8 total, 8 up, 8 in 2024-03-26T02:36:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:26 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3505804172' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T02:36:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:26 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T02:36:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:26 smithi072 ceph-mon[41619]: pool 'de5f38de-babb-481b-9bb5-ced33138378b' no longer out of quota; removing NO_QUOTA flag 2024-03-26T02:36:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:26 smithi072 ceph-mon[41619]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T02:36:26.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:26 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]': finished 2024-03-26T02:36:26.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:26 smithi072 ceph-mon[41619]: osdmap e832: 8 total, 8 up, 8 in 2024-03-26T02:36:27.031 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.031+0000 7f142affd640 1 -- 172.21.15.72:0/459206853 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "pg_num": 12}]=0 pool '6857fade-a9dd-4b5e-87a3-010f68d73cf2' created v833) v1 ==== 176+0+0 (secure 0 0 0) 0x7f143005a820 con 0x7f1434100450 2024-03-26T02:36:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:26 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]': finished 2024-03-26T02:36:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:26 smithi173 ceph-mon[33839]: osdmap e831: 8 total, 8 up, 8 in 2024-03-26T02:36:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:26 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3505804172' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T02:36:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:26 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T02:36:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:26 smithi173 ceph-mon[33839]: pool 'de5f38de-babb-481b-9bb5-ced33138378b' no longer out of quota; removing NO_QUOTA flag 2024-03-26T02:36:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:26 smithi173 ceph-mon[33839]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T02:36:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:26 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]': finished 2024-03-26T02:36:27.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:26 smithi173 ceph-mon[33839]: osdmap e832: 8 total, 8 up, 8 in 2024-03-26T02:36:27.142 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.142+0000 7f143bd48640 1 -- 172.21.15.72:0/459206853 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "pg_num": 12} v 0) v1 -- 0x7f1434068eb0 con 0x7f1434100450 2024-03-26T02:36:27.143 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.142+0000 7f142affd640 1 -- 172.21.15.72:0/459206853 <== mon.0 v2:172.21.15.72:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "pg_num": 12}]=0 pool '6857fade-a9dd-4b5e-87a3-010f68d73cf2' already exists v833) v1 ==== 183+0+0 (secure 0 0 0) 0x7f143005aa00 con 0x7f1434100450 2024-03-26T02:36:27.143 INFO:tasks.workunit.client.0.smithi072.stderr:pool '6857fade-a9dd-4b5e-87a3-010f68d73cf2' already exists 2024-03-26T02:36:27.145 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.144+0000 7f143bd48640 1 -- 172.21.15.72:0/459206853 >> v2:172.21.15.72:6800/3246690770 conn(0x7f142006d9f0 msgr2=0x7f142006feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:36:27.145 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.144+0000 7f143bd48640 1 --2- 172.21.15.72:0/459206853 >> v2:172.21.15.72:6800/3246690770 conn(0x7f142006d9f0 0x7f142006feb0 secure :-1 s=READY pgs=3389 cs=0 l=1 rev1=1 crypto rx=0x7f1434069db0 tx=0x7f142400b390 comp rx=0 tx=0).stop 2024-03-26T02:36:27.145 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.144+0000 7f143bd48640 1 -- 172.21.15.72:0/459206853 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1434100450 msgr2=0x7f1434195f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:36:27.145 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.144+0000 7f143bd48640 1 --2- 172.21.15.72:0/459206853 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1434100450 0x7f1434195f20 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f143000ea60 tx=0x7f143000ef20 comp rx=0 tx=0).stop 2024-03-26T02:36:27.145 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.144+0000 7f143bd48640 1 -- 172.21.15.72:0/459206853 shutdown_connections 2024-03-26T02:36:27.145 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.144+0000 7f143bd48640 1 --2- 172.21.15.72:0/459206853 >> v2:172.21.15.72:6800/3246690770 conn(0x7f142006d9f0 0x7f142006feb0 unknown :-1 s=CLOSED pgs=3389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:27.145 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.144+0000 7f143bd48640 1 --2- 172.21.15.72:0/459206853 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f14341064f0 0x7f14341969a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:27.145 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.144+0000 7f143bd48640 1 --2- 172.21.15.72:0/459206853 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f1434100e10 0x7f1434196460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:27.146 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.144+0000 7f143bd48640 1 --2- 172.21.15.72:0/459206853 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f1434100450 0x7f1434195f20 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:27.146 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.144+0000 7f143bd48640 1 -- 172.21.15.72:0/459206853 >> 172.21.15.72:0/459206853 conn(0x7f14340775b0 msgr2=0x7f14340ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:36:27.146 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.144+0000 7f143bd48640 1 -- 172.21.15.72:0/459206853 shutdown_connections 2024-03-26T02:36:27.146 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.144+0000 7f143bd48640 1 -- 172.21.15.72:0/459206853 wait complete. 2024-03-26T02:36:27.154 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph osd pool application enable 6857fade-a9dd-4b5e-87a3-010f68d73cf2 rados 2024-03-26T02:36:27.238 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.235+0000 7fa8e9fc9640 1 -- 172.21.15.72:0/1399795976 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa8e4101f40 msgr2=0x7fa8e410e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:36:27.238 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.235+0000 7fa8e9fc9640 1 --2- 172.21.15.72:0/1399795976 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa8e4101f40 0x7fa8e410e400 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7fa8dc009cd0 tx=0x7fa8dc02c0f0 comp rx=0 tx=0).stop 2024-03-26T02:36:27.238 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.235+0000 7fa8e9fc9640 1 -- 172.21.15.72:0/1399795976 shutdown_connections 2024-03-26T02:36:27.239 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.235+0000 7fa8e9fc9640 1 --2- 172.21.15.72:0/1399795976 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa8e4101f40 0x7fa8e410e400 unknown :-1 s=CLOSED pgs=2569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:27.239 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.235+0000 7fa8e9fc9640 1 --2- 172.21.15.72:0/1399795976 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8e4101580 0x7fa8e4101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:27.239 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.235+0000 7fa8e9fc9640 1 --2- 172.21.15.72:0/1399795976 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa8e4107640 0x7fa8e4107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:27.239 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.235+0000 7fa8e9fc9640 1 -- 172.21.15.72:0/1399795976 >> 172.21.15.72:0/1399795976 conn(0x7fa8e40fd090 msgr2=0x7fa8e40ff4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:36:27.239 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.235+0000 7fa8e9fc9640 1 -- 172.21.15.72:0/1399795976 shutdown_connections 2024-03-26T02:36:27.239 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.235+0000 7fa8e9fc9640 1 -- 172.21.15.72:0/1399795976 wait complete. 2024-03-26T02:36:27.239 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.235+0000 7fa8e9fc9640 1 Processor -- start 2024-03-26T02:36:27.239 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.236+0000 7fa8e9fc9640 1 -- start start 2024-03-26T02:36:27.239 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.236+0000 7fa8e9fc9640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa8e4101580 0x7fa8e41972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:36:27.239 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.236+0000 7fa8e9fc9640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa8e4101f40 0x7fa8e4197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:36:27.239 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.236+0000 7fa8e9fc9640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8e4107640 0x7fa8e419bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:36:27.239 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.236+0000 7fa8e9fc9640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e419c140 con 0x7fa8e4107640 2024-03-26T02:36:27.239 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.236+0000 7fa8e9fc9640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e419c2b0 con 0x7fa8e4101f40 2024-03-26T02:36:27.239 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.236+0000 7fa8e9fc9640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e419c470 con 0x7fa8e4101580 2024-03-26T02:36:27.240 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.236+0000 7fa8e37fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa8e4101580 0x7fa8e41972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:36:27.240 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.236+0000 7fa8e3fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8e4107640 0x7fa8e419bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:36:27.240 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.236+0000 7fa8e37fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa8e4101580 0x7fa8e41972d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:35948/0 (socket says 172.21.15.72:35948) 2024-03-26T02:36:27.240 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.236+0000 7fa8e37fe640 1 -- 172.21.15.72:0/3548379792 learned_addr learned my addr 172.21.15.72:0/3548379792 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:36:27.240 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.236+0000 7fa8e2ffd640 1 --2- 172.21.15.72:0/3548379792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa8e4101f40 0x7fa8e4197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:36:27.240 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.236+0000 7fa8e37fe640 1 -- 172.21.15.72:0/3548379792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa8e4101f40 msgr2=0x7fa8e4197810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:36:27.240 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.236+0000 7fa8e37fe640 1 --2- 172.21.15.72:0/3548379792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa8e4101f40 0x7fa8e4197810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:27.240 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.236+0000 7fa8e37fe640 1 -- 172.21.15.72:0/3548379792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8e4107640 msgr2=0x7fa8e419bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:36:27.240 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.236+0000 7fa8e37fe640 1 --2- 172.21.15.72:0/3548379792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8e4107640 0x7fa8e419bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:27.240 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.236+0000 7fa8e37fe640 1 -- 172.21.15.72:0/3548379792 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d8009490 con 0x7fa8e4101580 2024-03-26T02:36:27.240 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.236+0000 7fa8e3fff640 1 --2- 172.21.15.72:0/3548379792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8e4107640 0x7fa8e419bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:36:27.240 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.236+0000 7fa8e37fe640 1 --2- 172.21.15.72:0/3548379792 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa8e4101580 0x7fa8e41972d0 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7fa8d400b4a0 tx=0x7fa8d400b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:36:27.240 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.236+0000 7fa8e0ff9640 1 -- 172.21.15.72:0/3548379792 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d4010610 con 0x7fa8e4101580 2024-03-26T02:36:27.240 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.236+0000 7fa8e0ff9640 1 -- 172.21.15.72:0/3548379792 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fa8d4010c20 con 0x7fa8e4101580 2024-03-26T02:36:27.240 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.236+0000 7fa8e0ff9640 1 -- 172.21.15.72:0/3548379792 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d4017cd0 con 0x7fa8e4101580 2024-03-26T02:36:27.241 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.236+0000 7fa8e9fc9640 1 -- 172.21.15.72:0/3548379792 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8dc009900 con 0x7fa8e4101580 2024-03-26T02:36:27.241 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.236+0000 7fa8e9fc9640 1 -- 172.21.15.72:0/3548379792 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e41a3df0 con 0x7fa8e4101580 2024-03-26T02:36:27.241 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.237+0000 7fa8e9fc9640 1 -- 172.21.15.72:0/3548379792 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e40688c0 con 0x7fa8e4101580 2024-03-26T02:36:27.241 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.238+0000 7fa8e0ff9640 1 -- 172.21.15.72:0/3548379792 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7fa8d4010770 con 0x7fa8e4101580 2024-03-26T02:36:27.241 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.238+0000 7fa8e0ff9640 1 --2- 172.21.15.72:0/3548379792 >> v2:172.21.15.72:6800/3246690770 conn(0x7fa8cc06d9f0 0x7fa8cc06feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:36:27.241 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.238+0000 7fa8e0ff9640 1 -- 172.21.15.72:0/3548379792 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(833..833 src has 255..833) v4 ==== 8221+0+0 (secure 0 0 0) 0x7fa8d408d1e0 con 0x7fa8e4101580 2024-03-26T02:36:27.242 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.238+0000 7fa8e2ffd640 1 --2- 172.21.15.72:0/3548379792 >> v2:172.21.15.72:6800/3246690770 conn(0x7fa8cc06d9f0 0x7fa8cc06feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:36:27.242 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.239+0000 7fa8e2ffd640 1 --2- 172.21.15.72:0/3548379792 >> v2:172.21.15.72:6800/3246690770 conn(0x7fa8cc06d9f0 0x7fa8cc06feb0 secure :-1 s=READY pgs=3390 cs=0 l=1 rev1=1 crypto rx=0x7fa8e40690b0 tx=0x7fa8d8005f90 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:36:27.244 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.243+0000 7fa8e0ff9640 1 -- 172.21.15.72:0/3548379792 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa8d4059f40 con 0x7fa8e4101580 2024-03-26T02:36:27.420 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:27.419+0000 7fa8e9fc9640 1 -- 172.21.15.72:0/3548379792 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "app": "rados"} v 0) v1 -- 0x7fa8e4110dc0 con 0x7fa8e4101580 2024-03-26T02:36:27.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:27 smithi072 ceph-mon[30177]: pgmap v1802: 176 pgs: 176 active+clean; 721 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:27.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:27 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/459206853' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "pg_num": 12}]: dispatch 2024-03-26T02:36:27.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:27 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/459206853' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "pg_num": 12}]': finished 2024-03-26T02:36:27.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:27 smithi072 ceph-mon[30177]: osdmap e833: 8 total, 8 up, 8 in 2024-03-26T02:36:27.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:27 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/459206853' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "pg_num": 12}]: dispatch 2024-03-26T02:36:27.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:27 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3548379792' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "app": "rados"}]: dispatch 2024-03-26T02:36:27.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:27 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "app": "rados"}]: dispatch 2024-03-26T02:36:27.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:27 smithi072 ceph-mon[41619]: pgmap v1802: 176 pgs: 176 active+clean; 721 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:27.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:27 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/459206853' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "pg_num": 12}]: dispatch 2024-03-26T02:36:27.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:27 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/459206853' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "pg_num": 12}]': finished 2024-03-26T02:36:27.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:27 smithi072 ceph-mon[41619]: osdmap e833: 8 total, 8 up, 8 in 2024-03-26T02:36:27.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:27 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/459206853' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "pg_num": 12}]: dispatch 2024-03-26T02:36:27.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:27 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3548379792' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "app": "rados"}]: dispatch 2024-03-26T02:36:27.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:27 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "app": "rados"}]: dispatch 2024-03-26T02:36:28.038 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:28.037+0000 7fa8e0ff9640 1 -- 172.21.15.72:0/3548379792 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "app": "rados"}]=0 enabled application 'rados' on pool '6857fade-a9dd-4b5e-87a3-010f68d73cf2' v834) v1 ==== 213+0+0 (secure 0 0 0) 0x7fa8d4059ad0 con 0x7fa8e4101580 2024-03-26T02:36:28.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:27 smithi173 ceph-mon[33839]: pgmap v1802: 176 pgs: 176 active+clean; 721 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:28.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:27 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/459206853' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "pg_num": 12}]: dispatch 2024-03-26T02:36:28.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:27 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/459206853' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "pg_num": 12}]': finished 2024-03-26T02:36:28.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:27 smithi173 ceph-mon[33839]: osdmap e833: 8 total, 8 up, 8 in 2024-03-26T02:36:28.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:27 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/459206853' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "pg_num": 12}]: dispatch 2024-03-26T02:36:28.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:27 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3548379792' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "app": "rados"}]: dispatch 2024-03-26T02:36:28.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:27 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "app": "rados"}]: dispatch 2024-03-26T02:36:28.151 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:28.150+0000 7fa8e9fc9640 1 -- 172.21.15.72:0/3548379792 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "app": "rados"} v 0) v1 -- 0x7fa8e4110fd0 con 0x7fa8e4101580 2024-03-26T02:36:29.044 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.043+0000 7fa8e0ff9640 1 -- 172.21.15.72:0/3548379792 <== mon.2 v2:172.21.15.72:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "app": "rados"}]=0 enabled application 'rados' on pool '6857fade-a9dd-4b5e-87a3-010f68d73cf2' v835) v1 ==== 213+0+0 (secure 0 0 0) 0x7fa8d401e070 con 0x7fa8e4101580 2024-03-26T02:36:29.044 INFO:tasks.workunit.client.0.smithi072.stderr:enabled application 'rados' on pool '6857fade-a9dd-4b5e-87a3-010f68d73cf2' 2024-03-26T02:36:29.047 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.045+0000 7fa8e9fc9640 1 -- 172.21.15.72:0/3548379792 >> v2:172.21.15.72:6800/3246690770 conn(0x7fa8cc06d9f0 msgr2=0x7fa8cc06feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:36:29.047 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.045+0000 7fa8e9fc9640 1 --2- 172.21.15.72:0/3548379792 >> v2:172.21.15.72:6800/3246690770 conn(0x7fa8cc06d9f0 0x7fa8cc06feb0 secure :-1 s=READY pgs=3390 cs=0 l=1 rev1=1 crypto rx=0x7fa8e40690b0 tx=0x7fa8d8005f90 comp rx=0 tx=0).stop 2024-03-26T02:36:29.047 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.045+0000 7fa8e9fc9640 1 -- 172.21.15.72:0/3548379792 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa8e4101580 msgr2=0x7fa8e41972d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:36:29.047 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.045+0000 7fa8e9fc9640 1 --2- 172.21.15.72:0/3548379792 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa8e4101580 0x7fa8e41972d0 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7fa8d400b4a0 tx=0x7fa8d400b960 comp rx=0 tx=0).stop 2024-03-26T02:36:29.047 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.046+0000 7fa8e9fc9640 1 -- 172.21.15.72:0/3548379792 shutdown_connections 2024-03-26T02:36:29.047 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.046+0000 7fa8e9fc9640 1 --2- 172.21.15.72:0/3548379792 >> v2:172.21.15.72:6800/3246690770 conn(0x7fa8cc06d9f0 0x7fa8cc06feb0 unknown :-1 s=CLOSED pgs=3390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:29.047 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.046+0000 7fa8e9fc9640 1 --2- 172.21.15.72:0/3548379792 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa8e4107640 0x7fa8e419bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:29.047 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.046+0000 7fa8e9fc9640 1 --2- 172.21.15.72:0/3548379792 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa8e4101f40 0x7fa8e4197810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:29.047 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.046+0000 7fa8e9fc9640 1 --2- 172.21.15.72:0/3548379792 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa8e4101580 0x7fa8e41972d0 unknown :-1 s=CLOSED pgs=2570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:29.048 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.046+0000 7fa8e9fc9640 1 -- 172.21.15.72:0/3548379792 >> 172.21.15.72:0/3548379792 conn(0x7fa8e40fd090 msgr2=0x7fa8e4105130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:36:29.048 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.046+0000 7fa8e9fc9640 1 -- 172.21.15.72:0/3548379792 shutdown_connections 2024-03-26T02:36:29.048 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.046+0000 7fa8e9fc9640 1 -- 172.21.15.72:0/3548379792 wait complete. 2024-03-26T02:36:29.058 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph osd pool set-quota 6857fade-a9dd-4b5e-87a3-010f68d73cf2 max_objects 10 2024-03-26T02:36:29.143 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.142+0000 7fae35793640 1 -- 172.21.15.72:0/3847827733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae3010bbd0 msgr2=0x7fae3010e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:36:29.143 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.142+0000 7fae35793640 1 --2- 172.21.15.72:0/3847827733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae3010bbd0 0x7fae3010e000 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7fae28009be0 tx=0x7fae2802bf80 comp rx=0 tx=0).stop 2024-03-26T02:36:29.143 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.142+0000 7fae35793640 1 -- 172.21.15.72:0/3847827733 shutdown_connections 2024-03-26T02:36:29.143 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.142+0000 7fae35793640 1 --2- 172.21.15.72:0/3847827733 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae3010bbd0 0x7fae3010e000 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:29.143 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.142+0000 7fae35793640 1 --2- 172.21.15.72:0/3847827733 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae30103120 0x7fae3010b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:29.143 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.142+0000 7fae35793640 1 --2- 172.21.15.72:0/3847827733 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fae30102800 0x7fae30102be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:29.143 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.142+0000 7fae35793640 1 -- 172.21.15.72:0/3847827733 >> 172.21.15.72:0/3847827733 conn(0x7fae30077670 msgr2=0x7fae30077a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:36:29.144 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.142+0000 7fae35793640 1 -- 172.21.15.72:0/3847827733 shutdown_connections 2024-03-26T02:36:29.144 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.142+0000 7fae35793640 1 -- 172.21.15.72:0/3847827733 wait complete. 2024-03-26T02:36:29.146 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.143+0000 7fae35793640 1 Processor -- start 2024-03-26T02:36:29.146 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.143+0000 7fae35793640 1 -- start start 2024-03-26T02:36:29.146 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.143+0000 7fae35793640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fae30102800 0x7fae30197280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:36:29.146 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.143+0000 7fae35793640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae30103120 0x7fae301977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:36:29.147 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.143+0000 7fae35793640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae3010bbd0 0x7fae3019bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:36:29.147 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.143+0000 7fae35793640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3019c140 con 0x7fae30103120 2024-03-26T02:36:29.147 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.143+0000 7fae35793640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3019c2b0 con 0x7fae3010bbd0 2024-03-26T02:36:29.147 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.143+0000 7fae35793640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae3019c470 con 0x7fae30102800 2024-03-26T02:36:29.147 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.143+0000 7fae2effd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fae30102800 0x7fae30197280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:36:29.147 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.143+0000 7fae27fff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae30103120 0x7fae301977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:36:29.147 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.143+0000 7fae2effd640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fae30102800 0x7fae30197280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:35956/0 (socket says 172.21.15.72:35956) 2024-03-26T02:36:29.147 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.143+0000 7fae2effd640 1 -- 172.21.15.72:0/3983075652 learned_addr learned my addr 172.21.15.72:0/3983075652 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:36:29.147 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.143+0000 7fae2f7fe640 1 --2- 172.21.15.72:0/3983075652 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae3010bbd0 0x7fae3019bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:36:29.147 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.143+0000 7fae2effd640 1 -- 172.21.15.72:0/3983075652 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae3010bbd0 msgr2=0x7fae3019bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:36:29.147 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.143+0000 7fae2effd640 1 --2- 172.21.15.72:0/3983075652 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae3010bbd0 0x7fae3019bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:29.147 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.143+0000 7fae2effd640 1 -- 172.21.15.72:0/3983075652 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae30103120 msgr2=0x7fae301977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:36:29.147 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.143+0000 7fae2effd640 1 --2- 172.21.15.72:0/3983075652 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae30103120 0x7fae301977c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:29.147 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.143+0000 7fae2effd640 1 -- 172.21.15.72:0/3983075652 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae20009560 con 0x7fae30102800 2024-03-26T02:36:29.147 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.143+0000 7fae27fff640 1 --2- 172.21.15.72:0/3983075652 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae30103120 0x7fae301977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:36:29.148 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.144+0000 7fae2effd640 1 --2- 172.21.15.72:0/3983075652 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fae30102800 0x7fae30197280 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7fae2000e9c0 tx=0x7fae2000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:36:29.148 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.144+0000 7fae2cff9640 1 -- 172.21.15.72:0/3983075652 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2000cde0 con 0x7fae30102800 2024-03-26T02:36:29.148 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.144+0000 7fae2cff9640 1 -- 172.21.15.72:0/3983075652 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fae200044d0 con 0x7fae30102800 2024-03-26T02:36:29.148 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.144+0000 7fae35793640 1 -- 172.21.15.72:0/3983075652 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae28009840 con 0x7fae30102800 2024-03-26T02:36:29.148 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.144+0000 7fae2cff9640 1 -- 172.21.15.72:0/3983075652 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae20010430 con 0x7fae30102800 2024-03-26T02:36:29.148 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.144+0000 7fae35793640 1 -- 172.21.15.72:0/3983075652 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae30100380 con 0x7fae30102800 2024-03-26T02:36:29.148 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.145+0000 7fae35793640 1 -- 172.21.15.72:0/3983075652 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae300688c0 con 0x7fae30102800 2024-03-26T02:36:29.148 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.145+0000 7fae2cff9640 1 -- 172.21.15.72:0/3983075652 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7fae200035f0 con 0x7fae30102800 2024-03-26T02:36:29.148 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.145+0000 7fae2cff9640 1 --2- 172.21.15.72:0/3983075652 >> v2:172.21.15.72:6800/3246690770 conn(0x7fae1c06d9f0 0x7fae1c06feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:36:29.148 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.146+0000 7fae2cff9640 1 -- 172.21.15.72:0/3983075652 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(835..835 src has 255..835) v4 ==== 8234+0+0 (secure 0 0 0) 0x7fae20014070 con 0x7fae30102800 2024-03-26T02:36:29.148 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.146+0000 7fae27fff640 1 --2- 172.21.15.72:0/3983075652 >> v2:172.21.15.72:6800/3246690770 conn(0x7fae1c06d9f0 0x7fae1c06feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:36:29.149 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.146+0000 7fae27fff640 1 --2- 172.21.15.72:0/3983075652 >> v2:172.21.15.72:6800/3246690770 conn(0x7fae1c06d9f0 0x7fae1c06feb0 secure :-1 s=READY pgs=3391 cs=0 l=1 rev1=1 crypto rx=0x7fae1800ffd0 tx=0x7fae1800f7a0 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:36:29.151 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.150+0000 7fae2cff9640 1 -- 172.21.15.72:0/3983075652 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fae2005aca0 con 0x7fae30102800 2024-03-26T02:36:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:29 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "app": "rados"}]': finished 2024-03-26T02:36:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:29 smithi173 ceph-mon[33839]: osdmap e834: 8 total, 8 up, 8 in 2024-03-26T02:36:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:29 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3548379792' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "app": "rados"}]: dispatch 2024-03-26T02:36:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:29 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "app": "rados"}]: dispatch 2024-03-26T02:36:29.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:29 smithi173 ceph-mon[33839]: pgmap v1805: 188 pgs: 12 unknown, 176 active+clean; 721 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T02:36:29.325 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:29.324+0000 7fae35793640 1 -- 172.21.15.72:0/3983075652 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7fae301985d0 con 0x7fae30102800 2024-03-26T02:36:29.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:29 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "app": "rados"}]': finished 2024-03-26T02:36:29.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:29 smithi072 ceph-mon[30177]: osdmap e834: 8 total, 8 up, 8 in 2024-03-26T02:36:29.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:29 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3548379792' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "app": "rados"}]: dispatch 2024-03-26T02:36:29.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:29 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "app": "rados"}]: dispatch 2024-03-26T02:36:29.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:29 smithi072 ceph-mon[30177]: pgmap v1805: 188 pgs: 12 unknown, 176 active+clean; 721 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T02:36:29.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:29 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "app": "rados"}]': finished 2024-03-26T02:36:29.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:29 smithi072 ceph-mon[41619]: osdmap e834: 8 total, 8 up, 8 in 2024-03-26T02:36:29.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:29 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3548379792' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "app": "rados"}]: dispatch 2024-03-26T02:36:29.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:29 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "app": "rados"}]: dispatch 2024-03-26T02:36:29.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:29 smithi072 ceph-mon[41619]: pgmap v1805: 188 pgs: 12 unknown, 176 active+clean; 721 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T02:36:30.053 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:30.052+0000 7fae2cff9640 1 -- 172.21.15.72:0/3983075652 <== mon.2 v2:172.21.15.72:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 6857fade-a9dd-4b5e-87a3-010f68d73cf2 v836) v1 ==== 223+0+0 (secure 0 0 0) 0x7fae2005a830 con 0x7fae30102800 2024-03-26T02:36:30.160 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:30.160+0000 7fae35793640 1 -- 172.21.15.72:0/3983075652 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7fae301063d0 con 0x7fae30102800 2024-03-26T02:36:30.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "app": "rados"}]': finished 2024-03-26T02:36:30.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:30 smithi173 ceph-mon[33839]: osdmap e835: 8 total, 8 up, 8 in 2024-03-26T02:36:30.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:30 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3983075652' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T02:36:30.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:30 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T02:36:30.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "app": "rados"}]': finished 2024-03-26T02:36:30.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:30 smithi072 ceph-mon[30177]: osdmap e835: 8 total, 8 up, 8 in 2024-03-26T02:36:30.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:30 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3983075652' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T02:36:30.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:30 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T02:36:30.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "app": "rados"}]': finished 2024-03-26T02:36:30.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:30 smithi072 ceph-mon[41619]: osdmap e835: 8 total, 8 up, 8 in 2024-03-26T02:36:30.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:30 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3983075652' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T02:36:30.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:30 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T02:36:31.063 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:31.062+0000 7fae2cff9640 1 -- 172.21.15.72:0/3983075652 <== mon.2 v2:172.21.15.72:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 6857fade-a9dd-4b5e-87a3-010f68d73cf2 v837) v1 ==== 223+0+0 (secure 0 0 0) 0x7fae2005aa10 con 0x7fae30102800 2024-03-26T02:36:31.063 INFO:tasks.workunit.client.0.smithi072.stderr:set-quota max_objects = 10 for pool 6857fade-a9dd-4b5e-87a3-010f68d73cf2 2024-03-26T02:36:31.065 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:31.064+0000 7fae35793640 1 -- 172.21.15.72:0/3983075652 >> v2:172.21.15.72:6800/3246690770 conn(0x7fae1c06d9f0 msgr2=0x7fae1c06feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:36:31.066 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:31.064+0000 7fae35793640 1 --2- 172.21.15.72:0/3983075652 >> v2:172.21.15.72:6800/3246690770 conn(0x7fae1c06d9f0 0x7fae1c06feb0 secure :-1 s=READY pgs=3391 cs=0 l=1 rev1=1 crypto rx=0x7fae1800ffd0 tx=0x7fae1800f7a0 comp rx=0 tx=0).stop 2024-03-26T02:36:31.066 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:31.064+0000 7fae35793640 1 -- 172.21.15.72:0/3983075652 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fae30102800 msgr2=0x7fae30197280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:36:31.066 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:31.064+0000 7fae35793640 1 --2- 172.21.15.72:0/3983075652 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fae30102800 0x7fae30197280 secure :-1 s=READY pgs=2571 cs=0 l=1 rev1=1 crypto rx=0x7fae2000e9c0 tx=0x7fae2000ee80 comp rx=0 tx=0).stop 2024-03-26T02:36:31.066 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:31.064+0000 7fae35793640 1 -- 172.21.15.72:0/3983075652 shutdown_connections 2024-03-26T02:36:31.066 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:31.064+0000 7fae35793640 1 --2- 172.21.15.72:0/3983075652 >> v2:172.21.15.72:6800/3246690770 conn(0x7fae1c06d9f0 0x7fae1c06feb0 unknown :-1 s=CLOSED pgs=3391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:31.066 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:31.064+0000 7fae35793640 1 --2- 172.21.15.72:0/3983075652 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fae3010bbd0 0x7fae3019bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:31.066 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:31.064+0000 7fae35793640 1 --2- 172.21.15.72:0/3983075652 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fae30103120 0x7fae301977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:31.066 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:31.064+0000 7fae35793640 1 --2- 172.21.15.72:0/3983075652 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fae30102800 0x7fae30197280 unknown :-1 s=CLOSED pgs=2571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:36:31.066 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:31.064+0000 7fae35793640 1 -- 172.21.15.72:0/3983075652 >> 172.21.15.72:0/3983075652 conn(0x7fae30077670 msgr2=0x7fae301035a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:36:31.066 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:31.065+0000 7fae35793640 1 -- 172.21.15.72:0/3983075652 shutdown_connections 2024-03-26T02:36:31.066 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:36:31.065+0000 7fae35793640 1 -- 172.21.15.72:0/3983075652 wait complete. 2024-03-26T02:36:31.077 INFO:tasks.workunit.client.0.smithi072.stderr:+ sleep 30 2024-03-26T02:36:31.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:31 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "field": "max_objects", "val": "10"}]': finished 2024-03-26T02:36:31.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:31 smithi173 ceph-mon[33839]: osdmap e836: 8 total, 8 up, 8 in 2024-03-26T02:36:31.313 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:31 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3983075652' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T02:36:31.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:31 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T02:36:31.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:31 smithi173 ceph-mon[33839]: pgmap v1808: 188 pgs: 12 unknown, 176 active+clean; 721 KiB data, 899 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:36:31.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:31 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "field": "max_objects", "val": "10"}]': finished 2024-03-26T02:36:31.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:31 smithi072 ceph-mon[30177]: osdmap e836: 8 total, 8 up, 8 in 2024-03-26T02:36:31.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:31 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3983075652' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T02:36:31.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:31 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T02:36:31.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:31 smithi072 ceph-mon[30177]: pgmap v1808: 188 pgs: 12 unknown, 176 active+clean; 721 KiB data, 899 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:36:31.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:31 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "field": "max_objects", "val": "10"}]': finished 2024-03-26T02:36:31.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:31 smithi072 ceph-mon[41619]: osdmap e836: 8 total, 8 up, 8 in 2024-03-26T02:36:31.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:31 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3983075652' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T02:36:31.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:31 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T02:36:31.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:31 smithi072 ceph-mon[41619]: pgmap v1808: 188 pgs: 12 unknown, 176 active+clean; 721 KiB data, 899 MiB used, 714 GiB / 715 GiB avail 2024-03-26T02:36:32.314 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:36:32 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:36:32.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:32 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "field": "max_objects", "val": "10"}]': finished 2024-03-26T02:36:32.314 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:32 smithi173 ceph-mon[33839]: osdmap e837: 8 total, 8 up, 8 in 2024-03-26T02:36:32.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:32 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "field": "max_objects", "val": "10"}]': finished 2024-03-26T02:36:32.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:32 smithi072 ceph-mon[30177]: osdmap e837: 8 total, 8 up, 8 in 2024-03-26T02:36:32.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:32 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "field": "max_objects", "val": "10"}]': finished 2024-03-26T02:36:32.469 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:32 smithi072 ceph-mon[41619]: osdmap e837: 8 total, 8 up, 8 in 2024-03-26T02:36:33.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:33 smithi072 ceph-mon[30177]: pgmap v1810: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.8 KiB/s wr, 1 op/s 2024-03-26T02:36:33.469 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:33 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:33.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:33 smithi072 ceph-mon[41619]: pgmap v1810: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.8 KiB/s wr, 1 op/s 2024-03-26T02:36:33.470 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:33 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:33 smithi173 ceph-mon[33839]: pgmap v1810: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.8 KiB/s wr, 1 op/s 2024-03-26T02:36:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:33 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:35.248 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:36:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:36:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:36:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:35 smithi173 ceph-mon[33839]: pgmap v1811: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-26T02:36:35.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:35 smithi072 ceph-mon[30177]: pgmap v1811: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-26T02:36:35.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:35 smithi072 ceph-mon[41619]: pgmap v1811: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 4.0 KiB/s wr, 1 op/s 2024-03-26T02:36:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:37 smithi173 ceph-mon[33839]: pgmap v1812: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 3.4 KiB/s wr, 1 op/s 2024-03-26T02:36:37.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:37 smithi072 ceph-mon[30177]: pgmap v1812: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 3.4 KiB/s wr, 1 op/s 2024-03-26T02:36:37.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:37 smithi072 ceph-mon[41619]: pgmap v1812: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 3.4 KiB/s wr, 1 op/s 2024-03-26T02:36:39.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:39 smithi072 ceph-mon[30177]: pgmap v1813: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.9 KiB/s wr, 1 op/s 2024-03-26T02:36:39.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:39 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:36:39.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:39 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:36:39.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:39 smithi072 ceph-mon[41619]: pgmap v1813: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.9 KiB/s wr, 1 op/s 2024-03-26T02:36:39.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:39 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:36:39.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:39 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:36:39.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:39 smithi173 ceph-mon[33839]: pgmap v1813: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.9 KiB/s wr, 1 op/s 2024-03-26T02:36:39.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:39 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:36:39.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:39 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:36:41.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:41 smithi072 ceph-mon[30177]: pgmap v1814: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.4 KiB/s wr, 1 op/s 2024-03-26T02:36:41.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:41 smithi072 ceph-mon[41619]: pgmap v1814: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.4 KiB/s wr, 1 op/s 2024-03-26T02:36:41.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:41 smithi173 ceph-mon[33839]: pgmap v1814: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.4 KiB/s wr, 1 op/s 2024-03-26T02:36:42.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:36:42 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:36:43.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:43 smithi072 ceph-mon[30177]: pgmap v1815: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 919 B/s rd, 0 op/s 2024-03-26T02:36:43.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:43 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:43.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:43 smithi072 ceph-mon[41619]: pgmap v1815: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 919 B/s rd, 0 op/s 2024-03-26T02:36:43.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:43 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:43.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:43 smithi173 ceph-mon[33839]: pgmap v1815: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 919 B/s rd, 0 op/s 2024-03-26T02:36:43.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:43 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:45.399 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:36:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:36:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:36:45.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:45 smithi072 ceph-mon[30177]: pgmap v1816: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:45.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:45 smithi072 ceph-mon[41619]: pgmap v1816: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:45.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:45 smithi173 ceph-mon[33839]: pgmap v1816: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:47.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:47 smithi072 ceph-mon[30177]: pgmap v1817: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:47.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:47 smithi072 ceph-mon[41619]: pgmap v1817: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:47.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:47 smithi173 ceph-mon[33839]: pgmap v1817: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:49.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:49 smithi072 ceph-mon[30177]: pgmap v1818: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:49.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:49 smithi072 ceph-mon[41619]: pgmap v1818: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:49.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:49 smithi173 ceph-mon[33839]: pgmap v1818: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:51.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:51 smithi072 ceph-mon[30177]: pgmap v1819: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:51.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:51 smithi072 ceph-mon[41619]: pgmap v1819: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:51.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:51 smithi173 ceph-mon[33839]: pgmap v1819: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:52.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:36:52 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:36:53.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:53 smithi072 ceph-mon[30177]: pgmap v1820: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:53.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:53 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:53.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:53 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:36:53.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:53 smithi072 ceph-mon[41619]: pgmap v1820: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:53.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:53 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:53.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:53 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:36:53.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:53 smithi173 ceph-mon[33839]: pgmap v1820: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:53.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:53 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:36:53.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:53 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:36:55.416 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:36:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:36:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:36:55.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:55 smithi072 ceph-mon[41619]: pgmap v1821: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:55.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:55 smithi072 ceph-mon[30177]: pgmap v1821: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:55.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:55 smithi173 ceph-mon[33839]: pgmap v1821: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:57.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:57 smithi072 ceph-mon[30177]: pgmap v1822: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:57.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:57 smithi072 ceph-mon[41619]: pgmap v1822: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:57.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:57 smithi173 ceph-mon[33839]: pgmap v1822: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:36:59.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:36:59 smithi072 ceph-mon[30177]: pgmap v1823: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:59.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:36:59 smithi072 ceph-mon[41619]: pgmap v1823: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:36:59.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:36:59 smithi173 ceph-mon[33839]: pgmap v1823: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:01.078 INFO:tasks.workunit.client.0.smithi072.stderr:++ seq 1 10 2024-03-26T02:37:01.079 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in `seq 1 10` 2024-03-26T02:37:01.079 INFO:tasks.workunit.client.0.smithi072.stderr:+ rados -p 6857fade-a9dd-4b5e-87a3-010f68d73cf2 put obj1 /etc/passwd 2024-03-26T02:37:01.120 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in `seq 1 10` 2024-03-26T02:37:01.120 INFO:tasks.workunit.client.0.smithi072.stderr:+ rados -p 6857fade-a9dd-4b5e-87a3-010f68d73cf2 put obj2 /etc/passwd 2024-03-26T02:37:01.157 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in `seq 1 10` 2024-03-26T02:37:01.157 INFO:tasks.workunit.client.0.smithi072.stderr:+ rados -p 6857fade-a9dd-4b5e-87a3-010f68d73cf2 put obj3 /etc/passwd 2024-03-26T02:37:01.192 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in `seq 1 10` 2024-03-26T02:37:01.192 INFO:tasks.workunit.client.0.smithi072.stderr:+ rados -p 6857fade-a9dd-4b5e-87a3-010f68d73cf2 put obj4 /etc/passwd 2024-03-26T02:37:01.227 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in `seq 1 10` 2024-03-26T02:37:01.227 INFO:tasks.workunit.client.0.smithi072.stderr:+ rados -p 6857fade-a9dd-4b5e-87a3-010f68d73cf2 put obj5 /etc/passwd 2024-03-26T02:37:01.262 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in `seq 1 10` 2024-03-26T02:37:01.262 INFO:tasks.workunit.client.0.smithi072.stderr:+ rados -p 6857fade-a9dd-4b5e-87a3-010f68d73cf2 put obj6 /etc/passwd 2024-03-26T02:37:01.296 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in `seq 1 10` 2024-03-26T02:37:01.296 INFO:tasks.workunit.client.0.smithi072.stderr:+ rados -p 6857fade-a9dd-4b5e-87a3-010f68d73cf2 put obj7 /etc/passwd 2024-03-26T02:37:01.330 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in `seq 1 10` 2024-03-26T02:37:01.330 INFO:tasks.workunit.client.0.smithi072.stderr:+ rados -p 6857fade-a9dd-4b5e-87a3-010f68d73cf2 put obj8 /etc/passwd 2024-03-26T02:37:01.364 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in `seq 1 10` 2024-03-26T02:37:01.365 INFO:tasks.workunit.client.0.smithi072.stderr:+ rados -p 6857fade-a9dd-4b5e-87a3-010f68d73cf2 put obj9 /etc/passwd 2024-03-26T02:37:01.399 INFO:tasks.workunit.client.0.smithi072.stderr:+ for f in `seq 1 10` 2024-03-26T02:37:01.399 INFO:tasks.workunit.client.0.smithi072.stderr:+ rados -p 6857fade-a9dd-4b5e-87a3-010f68d73cf2 put obj10 /etc/passwd 2024-03-26T02:37:01.437 INFO:tasks.workunit.client.0.smithi072.stderr:+ sleep 30 2024-03-26T02:37:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:01 smithi072 ceph-mon[30177]: pgmap v1824: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:01.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:01 smithi072 ceph-mon[41619]: pgmap v1824: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:01.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:01 smithi173 ceph-mon[33839]: pgmap v1824: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:02.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:37:02 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:37:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:03 smithi072 ceph-mon[30177]: pgmap v1825: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:03 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:03.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:03 smithi072 ceph-mon[41619]: pgmap v1825: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:03.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:03 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:03 smithi173 ceph-mon[33839]: pgmap v1825: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:03 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:05.432 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:37:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:37:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:37:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:05 smithi072 ceph-mon[30177]: pgmap v1826: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:05.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:05 smithi072 ceph-mon[41619]: pgmap v1826: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:05.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:05 smithi173 ceph-mon[33839]: pgmap v1826: 188 pgs: 188 active+clean; 744 KiB data, 899 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:07.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:07 smithi072 ceph-mon[30177]: pgmap v1827: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-26T02:37:07.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:07 smithi072 ceph-mon[41619]: pgmap v1827: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-26T02:37:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:07 smithi173 ceph-mon[33839]: pgmap v1827: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-26T02:37:08.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:08 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:37:08.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:08 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:37:08.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:08 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:37:09.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:09 smithi072 ceph-mon[30177]: pgmap v1828: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 10 KiB/s wr, 1 op/s 2024-03-26T02:37:09.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:09 smithi072 ceph-mon[41619]: pgmap v1828: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 10 KiB/s wr, 1 op/s 2024-03-26T02:37:09.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:09 smithi173 ceph-mon[33839]: pgmap v1828: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 10 KiB/s wr, 1 op/s 2024-03-26T02:37:11.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:11 smithi072 ceph-mon[30177]: pgmap v1829: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 10 KiB/s wr, 1 op/s 2024-03-26T02:37:11.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:11 smithi072 ceph-mon[30177]: pool '6857fade-a9dd-4b5e-87a3-010f68d73cf2' is full (reached quota's max_objects: 10) 2024-03-26T02:37:11.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:11 smithi072 ceph-mon[30177]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T02:37:11.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:11 smithi072 ceph-mon[30177]: osdmap e838: 8 total, 8 up, 8 in 2024-03-26T02:37:11.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:11 smithi072 ceph-mon[41619]: pgmap v1829: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 10 KiB/s wr, 1 op/s 2024-03-26T02:37:11.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:11 smithi072 ceph-mon[41619]: pool '6857fade-a9dd-4b5e-87a3-010f68d73cf2' is full (reached quota's max_objects: 10) 2024-03-26T02:37:11.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:11 smithi072 ceph-mon[41619]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T02:37:11.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:11 smithi072 ceph-mon[41619]: osdmap e838: 8 total, 8 up, 8 in 2024-03-26T02:37:11.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:11 smithi173 ceph-mon[33839]: pgmap v1829: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 10 KiB/s wr, 1 op/s 2024-03-26T02:37:11.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:11 smithi173 ceph-mon[33839]: pool '6857fade-a9dd-4b5e-87a3-010f68d73cf2' is full (reached quota's max_objects: 10) 2024-03-26T02:37:11.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:11 smithi173 ceph-mon[33839]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T02:37:11.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:11 smithi173 ceph-mon[33839]: osdmap e838: 8 total, 8 up, 8 in 2024-03-26T02:37:12.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:37:12 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:37:13.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:13 smithi072 ceph-mon[41619]: pgmap v1831: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T02:37:13.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:13 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:13.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:13 smithi072 ceph-mon[30177]: pgmap v1831: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T02:37:13.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:13 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:13.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:13 smithi173 ceph-mon[33839]: pgmap v1831: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T02:37:13.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:13 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:15.456 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:37:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:37:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:37:15.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:15 smithi072 ceph-mon[41619]: pgmap v1832: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T02:37:15.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:15 smithi072 ceph-mon[30177]: pgmap v1832: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T02:37:15.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:15 smithi173 ceph-mon[33839]: pgmap v1832: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T02:37:17.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:17 smithi072 ceph-mon[30177]: pgmap v1833: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:37:17.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:17 smithi072 ceph-mon[41619]: pgmap v1833: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:37:17.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:17 smithi173 ceph-mon[33839]: pgmap v1833: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:37:19.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:19 smithi072 ceph-mon[30177]: pgmap v1834: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:37:19.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:19 smithi072 ceph-mon[41619]: pgmap v1834: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:37:19.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:19 smithi173 ceph-mon[33839]: pgmap v1834: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:37:21.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:21 smithi072 ceph-mon[30177]: pgmap v1835: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:37:21.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:21 smithi072 ceph-mon[41619]: pgmap v1835: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:37:21.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:21 smithi173 ceph-mon[33839]: pgmap v1835: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T02:37:22.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:37:22 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:37:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:23 smithi072 ceph-mon[30177]: pgmap v1836: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-03-26T02:37:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:23 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:23 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:37:23.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:23 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:37:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:23 smithi072 ceph-mon[41619]: pgmap v1836: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-03-26T02:37:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:23 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:23 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:37:23.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:23 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:37:23.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:23 smithi173 ceph-mon[33839]: pgmap v1836: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 916 B/s rd, 0 op/s 2024-03-26T02:37:23.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:23 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:23.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:23 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:37:23.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:23 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:37:25.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:37:24 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:37:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:37:25.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:25 smithi173 ceph-mon[33839]: pgmap v1837: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:25.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:25 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:37:25.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:25 smithi072 ceph-mon[30177]: pgmap v1837: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:25.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:25 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:37:25.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:25 smithi072 ceph-mon[41619]: pgmap v1837: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:25.969 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:25 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T02:37:26.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:26 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:37:26.969 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:26 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:37:26.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:26 smithi072 ceph-mon[30177]: pgmap v1838: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:26.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:26 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:37:26.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:26 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:37:26.970 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:26 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:37:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:26 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:37:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:26 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:37:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:26 smithi072 ceph-mon[41619]: pgmap v1838: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:26.970 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:26 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:37:26.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:26 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:37:26.971 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:26 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:37:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:26 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:37:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:26 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:37:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:26 smithi173 ceph-mon[33839]: pgmap v1838: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:27.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:26 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T02:37:27.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:26 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T02:37:27.064 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:26 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' 2024-03-26T02:37:29.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:29 smithi173 ceph-mon[33839]: pgmap v1839: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:29.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:29 smithi072 ceph-mon[30177]: pgmap v1839: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:29.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:29 smithi072 ceph-mon[41619]: pgmap v1839: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:31.438 INFO:tasks.workunit.client.0.smithi072.stderr:+ rados -p de5f38de-babb-481b-9bb5-ced33138378b put threemore /etc/passwd 2024-03-26T02:37:31.478 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph osd pool set-quota de5f38de-babb-481b-9bb5-ced33138378b max_bytes 0 2024-03-26T02:37:31.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:31 smithi173 ceph-mon[33839]: pgmap v1840: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:31.569 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.566+0000 7fea19a9d640 1 -- 172.21.15.72:0/367902405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea14107640 msgr2=0x7fea14107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:37:31.570 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.566+0000 7fea19a9d640 1 --2- 172.21.15.72:0/367902405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea14107640 0x7fea14107a20 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7fea04009900 tx=0x7fea0402bc30 comp rx=0 tx=0).stop 2024-03-26T02:37:31.570 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.566+0000 7fea19a9d640 1 -- 172.21.15.72:0/367902405 shutdown_connections 2024-03-26T02:37:31.570 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.566+0000 7fea19a9d640 1 --2- 172.21.15.72:0/367902405 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea14101f40 0x7fea1410e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:37:31.570 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.566+0000 7fea19a9d640 1 --2- 172.21.15.72:0/367902405 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fea14101580 0x7fea14101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:37:31.570 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.566+0000 7fea19a9d640 1 --2- 172.21.15.72:0/367902405 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea14107640 0x7fea14107a20 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:37:31.570 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.566+0000 7fea19a9d640 1 -- 172.21.15.72:0/367902405 >> 172.21.15.72:0/367902405 conn(0x7fea140fd050 msgr2=0x7fea140ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:37:31.570 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.566+0000 7fea19a9d640 1 -- 172.21.15.72:0/367902405 shutdown_connections 2024-03-26T02:37:31.570 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.566+0000 7fea19a9d640 1 -- 172.21.15.72:0/367902405 wait complete. 2024-03-26T02:37:31.570 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.566+0000 7fea19a9d640 1 Processor -- start 2024-03-26T02:37:31.570 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.567+0000 7fea19a9d640 1 -- start start 2024-03-26T02:37:31.571 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.567+0000 7fea19a9d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea14101580 0x7fea14197290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:37:31.571 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.567+0000 7fea19a9d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea14101f40 0x7fea141977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:37:31.571 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.567+0000 7fea19a9d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fea14107640 0x7fea1419bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:37:31.571 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.567+0000 7fea19a9d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1419c150 con 0x7fea14101f40 2024-03-26T02:37:31.571 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.567+0000 7fea19a9d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1419c2c0 con 0x7fea14101580 2024-03-26T02:37:31.571 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.567+0000 7fea19a9d640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea1419c480 con 0x7fea14107640 2024-03-26T02:37:31.571 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.567+0000 7fea12ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea14101f40 0x7fea141977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:37:31.571 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.567+0000 7fea13fff640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fea14107640 0x7fea1419bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:37:31.571 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.567+0000 7fea12ffd640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea14101f40 0x7fea141977d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:54520/0 (socket says 172.21.15.72:54520) 2024-03-26T02:37:31.571 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.567+0000 7fea12ffd640 1 -- 172.21.15.72:0/1000522746 learned_addr learned my addr 172.21.15.72:0/1000522746 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:37:31.571 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.567+0000 7fea137fe640 1 --2- 172.21.15.72:0/1000522746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea14101580 0x7fea14197290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:37:31.571 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.567+0000 7fea12ffd640 1 -- 172.21.15.72:0/1000522746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fea14107640 msgr2=0x7fea1419bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:37:31.571 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.567+0000 7fea12ffd640 1 --2- 172.21.15.72:0/1000522746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fea14107640 0x7fea1419bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:37:31.571 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.567+0000 7fea12ffd640 1 -- 172.21.15.72:0/1000522746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea14101580 msgr2=0x7fea14197290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:37:31.572 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.567+0000 7fea12ffd640 1 --2- 172.21.15.72:0/1000522746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea14101580 0x7fea14197290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:37:31.572 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.567+0000 7fea12ffd640 1 -- 172.21.15.72:0/1000522746 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea04009560 con 0x7fea14101f40 2024-03-26T02:37:31.572 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.567+0000 7fea13fff640 1 --2- 172.21.15.72:0/1000522746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fea14107640 0x7fea1419bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:37:31.572 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.567+0000 7fea12ffd640 1 --2- 172.21.15.72:0/1000522746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea14101f40 0x7fea141977d0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7fea0800e9f0 tx=0x7fea0800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:37:31.572 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.567+0000 7fea10ff9640 1 -- 172.21.15.72:0/1000522746 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea08009720 con 0x7fea14101f40 2024-03-26T02:37:31.572 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.567+0000 7fea19a9d640 1 -- 172.21.15.72:0/1000522746 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea1419c760 con 0x7fea14101f40 2024-03-26T02:37:31.572 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.567+0000 7fea10ff9640 1 -- 172.21.15.72:0/1000522746 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fea0800cd80 con 0x7fea14101f40 2024-03-26T02:37:31.572 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.568+0000 7fea10ff9640 1 -- 172.21.15.72:0/1000522746 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea08010430 con 0x7fea14101f40 2024-03-26T02:37:31.572 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.568+0000 7fea19a9d640 1 -- 172.21.15.72:0/1000522746 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea141a3fe0 con 0x7fea14101f40 2024-03-26T02:37:31.572 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.569+0000 7fea19a9d640 1 -- 172.21.15.72:0/1000522746 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea140688c0 con 0x7fea14101f40 2024-03-26T02:37:31.572 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.569+0000 7fea10ff9640 1 -- 172.21.15.72:0/1000522746 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7fea08005020 con 0x7fea14101f40 2024-03-26T02:37:31.573 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.569+0000 7fea10ff9640 1 --2- 172.21.15.72:0/1000522746 >> v2:172.21.15.72:6800/3246690770 conn(0x7fe9fc06d9f0 0x7fe9fc06feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:37:31.573 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.570+0000 7fea137fe640 1 --2- 172.21.15.72:0/1000522746 >> v2:172.21.15.72:6800/3246690770 conn(0x7fe9fc06d9f0 0x7fe9fc06feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:37:31.573 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.570+0000 7fea10ff9640 1 -- 172.21.15.72:0/1000522746 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(838..838 src has 255..838) v4 ==== 8234+0+0 (secure 0 0 0) 0x7fea08014070 con 0x7fea14101f40 2024-03-26T02:37:31.573 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.570+0000 7fea10ff9640 1 -- 172.21.15.72:0/1000522746 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=839}) v3 -- 0x7fe9fc079780 con 0x7fea14101f40 2024-03-26T02:37:31.573 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.570+0000 7fea137fe640 1 --2- 172.21.15.72:0/1000522746 >> v2:172.21.15.72:6800/3246690770 conn(0x7fe9fc06d9f0 0x7fe9fc06feb0 secure :-1 s=READY pgs=3403 cs=0 l=1 rev1=1 crypto rx=0x7fea04006010 tx=0x7fea04000bc0 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:37:31.575 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.574+0000 7fea10ff9640 1 -- 172.21.15.72:0/1000522746 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fea0805ad40 con 0x7fea14101f40 2024-03-26T02:37:31.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:31 smithi072 ceph-mon[30177]: pgmap v1840: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:31.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:31 smithi072 ceph-mon[41619]: pgmap v1840: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:31.751 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:31.750+0000 7fea19a9d640 1 -- 172.21.15.72:0/1000522746 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7fea14110d60 con 0x7fea14101f40 2024-03-26T02:37:32.273 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:32.272+0000 7fea10ff9640 1 -- 172.21.15.72:0/1000522746 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool de5f38de-babb-481b-9bb5-ced33138378b v839) v1 ==== 217+0+0 (secure 0 0 0) 0x7fea0805a8d0 con 0x7fea14101f40 2024-03-26T02:37:32.274 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:32.274+0000 7fea10ff9640 1 -- 172.21.15.72:0/1000522746 <== mon.0 v2:172.21.15.72:3300/0 8 ==== osd_map(839..839 src has 255..839) v4 ==== 628+0+0 (secure 0 0 0) 0x7fea0805b670 con 0x7fea14101f40 2024-03-26T02:37:32.274 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:32.274+0000 7fea10ff9640 1 -- 172.21.15.72:0/1000522746 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=840}) v3 -- 0x7fe9fc07a6e0 con 0x7fea14101f40 2024-03-26T02:37:32.386 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:32.385+0000 7fea19a9d640 1 -- 172.21.15.72:0/1000522746 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7fea14110ef0 con 0x7fea14101f40 2024-03-26T02:37:32.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:32 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1000522746' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T02:37:32.563 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:37:32 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:37:32.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:32 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1000522746' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T02:37:32.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:32 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1000522746' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T02:37:33.284 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.283+0000 7fea10ff9640 1 -- 172.21.15.72:0/1000522746 <== mon.0 v2:172.21.15.72:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool de5f38de-babb-481b-9bb5-ced33138378b v840) v1 ==== 217+0+0 (secure 0 0 0) 0x7fea0805aab0 con 0x7fea14101f40 2024-03-26T02:37:33.284 INFO:tasks.workunit.client.0.smithi072.stderr:set-quota max_bytes = 0 for pool de5f38de-babb-481b-9bb5-ced33138378b 2024-03-26T02:37:33.287 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.286+0000 7fea19a9d640 1 -- 172.21.15.72:0/1000522746 >> v2:172.21.15.72:6800/3246690770 conn(0x7fe9fc06d9f0 msgr2=0x7fe9fc06feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:37:33.287 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.286+0000 7fea19a9d640 1 --2- 172.21.15.72:0/1000522746 >> v2:172.21.15.72:6800/3246690770 conn(0x7fe9fc06d9f0 0x7fe9fc06feb0 secure :-1 s=READY pgs=3403 cs=0 l=1 rev1=1 crypto rx=0x7fea04006010 tx=0x7fea04000bc0 comp rx=0 tx=0).stop 2024-03-26T02:37:33.287 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.286+0000 7fea19a9d640 1 -- 172.21.15.72:0/1000522746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea14101f40 msgr2=0x7fea141977d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:37:33.287 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.286+0000 7fea19a9d640 1 --2- 172.21.15.72:0/1000522746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea14101f40 0x7fea141977d0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7fea0800e9f0 tx=0x7fea0800edf0 comp rx=0 tx=0).stop 2024-03-26T02:37:33.288 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.286+0000 7fea10ff9640 1 -- 172.21.15.72:0/1000522746 <== mon.0 v2:172.21.15.72:3300/0 10 ==== osd_map(840..840 src has 255..840) v4 ==== 628+0+0 (secure 0 0 0) 0x7fea0805f0c0 con 0x7fea14101f40 2024-03-26T02:37:33.288 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.286+0000 7fea19a9d640 1 -- 172.21.15.72:0/1000522746 shutdown_connections 2024-03-26T02:37:33.288 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.286+0000 7fea19a9d640 1 --2- 172.21.15.72:0/1000522746 >> v2:172.21.15.72:6800/3246690770 conn(0x7fe9fc06d9f0 0x7fe9fc06feb0 unknown :-1 s=CLOSED pgs=3403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:37:33.288 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.286+0000 7fea19a9d640 1 --2- 172.21.15.72:0/1000522746 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fea14107640 0x7fea1419bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:37:33.288 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.286+0000 7fea19a9d640 1 --2- 172.21.15.72:0/1000522746 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fea14101f40 0x7fea141977d0 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:37:33.288 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.286+0000 7fea19a9d640 1 --2- 172.21.15.72:0/1000522746 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fea14101580 0x7fea14197290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:37:33.288 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.286+0000 7fea19a9d640 1 -- 172.21.15.72:0/1000522746 >> 172.21.15.72:0/1000522746 conn(0x7fea140fd050 msgr2=0x7fea140fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:37:33.288 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.286+0000 7fea19a9d640 1 -- 172.21.15.72:0/1000522746 shutdown_connections 2024-03-26T02:37:33.288 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.286+0000 7fea19a9d640 1 -- 172.21.15.72:0/1000522746 wait complete. 2024-03-26T02:37:33.302 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph osd pool set-quota de5f38de-babb-481b-9bb5-ced33138378b max_objects 0 2024-03-26T02:37:33.392 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.388+0000 7fa46189d640 1 -- 172.21.15.72:0/918387539 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa45c106500 msgr2=0x7fa45c1068e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:37:33.392 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.388+0000 7fa46189d640 1 --2- 172.21.15.72:0/918387539 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa45c106500 0x7fa45c1068e0 secure :-1 s=READY pgs=2588 cs=0 l=1 rev1=1 crypto rx=0x7fa44c009cd0 tx=0x7fa44c02c0f0 comp rx=0 tx=0).stop 2024-03-26T02:37:33.392 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.388+0000 7fa46189d640 1 -- 172.21.15.72:0/918387539 shutdown_connections 2024-03-26T02:37:33.392 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.388+0000 7fa46189d640 1 --2- 172.21.15.72:0/918387539 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa45c100e20 0x7fa45c10d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:37:33.392 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.388+0000 7fa46189d640 1 --2- 172.21.15.72:0/918387539 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa45c100460 0x7fa45c1008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:37:33.392 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.388+0000 7fa46189d640 1 --2- 172.21.15.72:0/918387539 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa45c106500 0x7fa45c1068e0 unknown :-1 s=CLOSED pgs=2588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:37:33.393 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.388+0000 7fa46189d640 1 -- 172.21.15.72:0/918387539 >> 172.21.15.72:0/918387539 conn(0x7fa45c077610 msgr2=0x7fa45c077a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:37:33.393 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.388+0000 7fa46189d640 1 -- 172.21.15.72:0/918387539 shutdown_connections 2024-03-26T02:37:33.393 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.388+0000 7fa46189d640 1 -- 172.21.15.72:0/918387539 wait complete. 2024-03-26T02:37:33.393 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.389+0000 7fa46189d640 1 Processor -- start 2024-03-26T02:37:33.393 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.389+0000 7fa46189d640 1 -- start start 2024-03-26T02:37:33.393 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.389+0000 7fa46189d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa45c100460 0x7fa45c1962b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:37:33.393 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.389+0000 7fa46189d640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa45c100e20 0x7fa45c1967f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:37:33.393 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.389+0000 7fa46189d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa45c106500 0x7fa45c19ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:37:33.393 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.389+0000 7fa46189d640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c19b170 con 0x7fa45c100e20 2024-03-26T02:37:33.393 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.389+0000 7fa46189d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c19b2e0 con 0x7fa45c100460 2024-03-26T02:37:33.393 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.389+0000 7fa46189d640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c19b4a0 con 0x7fa45c106500 2024-03-26T02:37:33.393 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.389+0000 7fa45a7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa45c100e20 0x7fa45c1967f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:37:33.393 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.389+0000 7fa45b7fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa45c106500 0x7fa45c19ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:37:33.394 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.389+0000 7fa45b7fe640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa45c106500 0x7fa45c19ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3301/0 says I am v2:172.21.15.72:56886/0 (socket says 172.21.15.72:56886) 2024-03-26T02:37:33.394 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.389+0000 7fa45b7fe640 1 -- 172.21.15.72:0/3233145617 learned_addr learned my addr 172.21.15.72:0/3233145617 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:37:33.394 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.389+0000 7fa45a7fc640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa45c100e20 0x7fa45c1967f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40944/0 (socket says 172.21.15.72:40944) 2024-03-26T02:37:33.394 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.390+0000 7fa45affd640 1 --2- 172.21.15.72:0/3233145617 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa45c100460 0x7fa45c1962b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:37:33.394 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.390+0000 7fa45b7fe640 1 -- 172.21.15.72:0/3233145617 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa45c100460 msgr2=0x7fa45c1962b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:37:33.394 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.390+0000 7fa45b7fe640 1 --2- 172.21.15.72:0/3233145617 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa45c100460 0x7fa45c1962b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:37:33.394 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.390+0000 7fa45b7fe640 1 -- 172.21.15.72:0/3233145617 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa45c100e20 msgr2=0x7fa45c1967f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:37:33.394 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.390+0000 7fa45b7fe640 1 --2- 172.21.15.72:0/3233145617 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa45c100e20 0x7fa45c1967f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:37:33.394 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.390+0000 7fa45b7fe640 1 -- 172.21.15.72:0/3233145617 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa450009490 con 0x7fa45c106500 2024-03-26T02:37:33.394 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.390+0000 7fa45affd640 1 --2- 172.21.15.72:0/3233145617 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa45c100460 0x7fa45c1962b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T02:37:33.394 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.390+0000 7fa45a7fc640 1 --2- 172.21.15.72:0/3233145617 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa45c100e20 0x7fa45c1967f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:37:33.394 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.390+0000 7fa45b7fe640 1 --2- 172.21.15.72:0/3233145617 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa45c106500 0x7fa45c19ab70 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7fa454009fd0 tx=0x7fa45400b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:37:33.394 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.390+0000 7fa46089b640 1 -- 172.21.15.72:0/3233145617 <== mon.2 v2:172.21.15.72:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa454011830 con 0x7fa45c106500 2024-03-26T02:37:33.394 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.390+0000 7fa46089b640 1 -- 172.21.15.72:0/3233145617 <== mon.2 v2:172.21.15.72:3301/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fa454011e40 con 0x7fa45c106500 2024-03-26T02:37:33.395 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.390+0000 7fa46089b640 1 -- 172.21.15.72:0/3233145617 <== mon.2 v2:172.21.15.72:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45400f2e0 con 0x7fa45c106500 2024-03-26T02:37:33.395 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.390+0000 7fa46189d640 1 -- 172.21.15.72:0/3233145617 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa44c009900 con 0x7fa45c106500 2024-03-26T02:37:33.395 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.390+0000 7fa46189d640 1 -- 172.21.15.72:0/3233145617 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa45c1a2e20 con 0x7fa45c106500 2024-03-26T02:37:33.395 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.391+0000 7fa46189d640 1 -- 172.21.15.72:0/3233145617 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa45c0688c0 con 0x7fa45c106500 2024-03-26T02:37:33.395 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.391+0000 7fa46089b640 1 -- 172.21.15.72:0/3233145617 <== mon.2 v2:172.21.15.72:3301/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7fa454011990 con 0x7fa45c106500 2024-03-26T02:37:33.396 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.392+0000 7fa46089b640 1 --2- 172.21.15.72:0/3233145617 >> v2:172.21.15.72:6800/3246690770 conn(0x7fa44406d9f0 0x7fa44406feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:37:33.396 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.392+0000 7fa46089b640 1 -- 172.21.15.72:0/3233145617 <== mon.2 v2:172.21.15.72:3301/0 5 ==== osd_map(840..840 src has 255..840) v4 ==== 8234+0+0 (secure 0 0 0) 0x7fa45408d390 con 0x7fa45c106500 2024-03-26T02:37:33.396 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.392+0000 7fa45affd640 1 --2- 172.21.15.72:0/3233145617 >> v2:172.21.15.72:6800/3246690770 conn(0x7fa44406d9f0 0x7fa44406feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:37:33.396 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.392+0000 7fa46089b640 1 -- 172.21.15.72:0/3233145617 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=841}) v3 -- 0x7fa444079800 con 0x7fa45c106500 2024-03-26T02:37:33.396 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.392+0000 7fa45affd640 1 --2- 172.21.15.72:0/3233145617 >> v2:172.21.15.72:6800/3246690770 conn(0x7fa44406d9f0 0x7fa44406feb0 secure :-1 s=READY pgs=3404 cs=0 l=1 rev1=1 crypto rx=0x7fa44c009430 tx=0x7fa44c005d80 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:37:33.397 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.396+0000 7fa46089b640 1 -- 172.21.15.72:0/3233145617 <== mon.2 v2:172.21.15.72:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fa45405a030 con 0x7fa45c106500 2024-03-26T02:37:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:33 smithi173 ceph-mon[33839]: pgmap v1841: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:33 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1000522746' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T02:37:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:33 smithi173 ceph-mon[33839]: osdmap e839: 8 total, 8 up, 8 in 2024-03-26T02:37:33.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:33 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:33.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:33 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1000522746' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T02:37:33.572 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:33.571+0000 7fa46189d640 1 -- 172.21.15.72:0/3233145617 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7fa45c10fcc0 con 0x7fa45c106500 2024-03-26T02:37:33.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:33 smithi072 ceph-mon[41619]: pgmap v1841: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:33.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:33 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1000522746' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T02:37:33.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:33 smithi072 ceph-mon[41619]: osdmap e839: 8 total, 8 up, 8 in 2024-03-26T02:37:33.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:33 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:33.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:33 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1000522746' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T02:37:33.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:33 smithi072 ceph-mon[30177]: pgmap v1841: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:33.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:33 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1000522746' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T02:37:33.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:33 smithi072 ceph-mon[30177]: osdmap e839: 8 total, 8 up, 8 in 2024-03-26T02:37:33.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:33 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:33.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:33 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1000522746' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T02:37:34.294 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:34.293+0000 7fa46089b640 1 -- 172.21.15.72:0/3233145617 <== mon.2 v2:172.21.15.72:3301/0 7 ==== osd_map(841..841 src has 255..841) v4 ==== 628+0+0 (secure 0 0 0) 0x7fa45405a960 con 0x7fa45c106500 2024-03-26T02:37:34.294 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:34.293+0000 7fa46089b640 1 -- 172.21.15.72:0/3233145617 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=842}) v3 -- 0x7fa44407a370 con 0x7fa45c106500 2024-03-26T02:37:34.301 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:34.300+0000 7fa46089b640 1 -- 172.21.15.72:0/3233145617 <== mon.2 v2:172.21.15.72:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool de5f38de-babb-481b-9bb5-ced33138378b v841) v1 ==== 221+0+0 (secure 0 0 0) 0x7fa454059bc0 con 0x7fa45c106500 2024-03-26T02:37:34.413 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:34.412+0000 7fa46189d640 1 -- 172.21.15.72:0/3233145617 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7fa45c10fe50 con 0x7fa45c106500 2024-03-26T02:37:34.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:34 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/1000522746' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T02:37:34.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:34 smithi173 ceph-mon[33839]: osdmap e840: 8 total, 8 up, 8 in 2024-03-26T02:37:34.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:34 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3233145617' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T02:37:34.564 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:34 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T02:37:34.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:34 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/1000522746' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T02:37:34.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:34 smithi072 ceph-mon[41619]: osdmap e840: 8 total, 8 up, 8 in 2024-03-26T02:37:34.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:34 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3233145617' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T02:37:34.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:34 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T02:37:34.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:34 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/1000522746' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T02:37:34.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:34 smithi072 ceph-mon[30177]: osdmap e840: 8 total, 8 up, 8 in 2024-03-26T02:37:34.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:34 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3233145617' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T02:37:34.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:34 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T02:37:35.298 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:35.298+0000 7fa46089b640 1 -- 172.21.15.72:0/3233145617 <== mon.2 v2:172.21.15.72:3301/0 9 ==== osd_map(842..842 src has 255..842) v4 ==== 628+0+0 (secure 0 0 0) 0x7fa454059bc0 con 0x7fa45c106500 2024-03-26T02:37:35.299 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:35.298+0000 7fa46089b640 1 -- 172.21.15.72:0/3233145617 --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_subscribe({osdmap=843}) v3 -- 0x7fa44407a7e0 con 0x7fa45c106500 2024-03-26T02:37:35.304 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:35.303+0000 7fa46089b640 1 -- 172.21.15.72:0/3233145617 <== mon.2 v2:172.21.15.72:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool de5f38de-babb-481b-9bb5-ced33138378b v842) v1 ==== 221+0+0 (secure 0 0 0) 0x7fa454059da0 con 0x7fa45c106500 2024-03-26T02:37:35.304 INFO:tasks.workunit.client.0.smithi072.stderr:set-quota max_objects = 0 for pool de5f38de-babb-481b-9bb5-ced33138378b 2024-03-26T02:37:35.306 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:37:34 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:37:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:37:35.308 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:35.307+0000 7fa46189d640 1 -- 172.21.15.72:0/3233145617 >> v2:172.21.15.72:6800/3246690770 conn(0x7fa44406d9f0 msgr2=0x7fa44406feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:37:35.308 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:35.307+0000 7fa46189d640 1 --2- 172.21.15.72:0/3233145617 >> v2:172.21.15.72:6800/3246690770 conn(0x7fa44406d9f0 0x7fa44406feb0 secure :-1 s=READY pgs=3404 cs=0 l=1 rev1=1 crypto rx=0x7fa44c009430 tx=0x7fa44c005d80 comp rx=0 tx=0).stop 2024-03-26T02:37:35.308 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:35.307+0000 7fa46189d640 1 -- 172.21.15.72:0/3233145617 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa45c106500 msgr2=0x7fa45c19ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:37:35.308 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:35.307+0000 7fa46189d640 1 --2- 172.21.15.72:0/3233145617 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa45c106500 0x7fa45c19ab70 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7fa454009fd0 tx=0x7fa45400b8e0 comp rx=0 tx=0).stop 2024-03-26T02:37:35.308 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:35.307+0000 7fa46189d640 1 -- 172.21.15.72:0/3233145617 shutdown_connections 2024-03-26T02:37:35.308 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:35.307+0000 7fa46189d640 1 --2- 172.21.15.72:0/3233145617 >> v2:172.21.15.72:6800/3246690770 conn(0x7fa44406d9f0 0x7fa44406feb0 unknown :-1 s=CLOSED pgs=3404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:37:35.309 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:35.307+0000 7fa46189d640 1 --2- 172.21.15.72:0/3233145617 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fa45c106500 0x7fa45c19ab70 unknown :-1 s=CLOSED pgs=2589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:37:35.309 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:35.307+0000 7fa46189d640 1 --2- 172.21.15.72:0/3233145617 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fa45c100e20 0x7fa45c1967f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:37:35.309 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:35.307+0000 7fa46189d640 1 --2- 172.21.15.72:0/3233145617 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa45c100460 0x7fa45c1962b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:37:35.309 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:35.307+0000 7fa46189d640 1 -- 172.21.15.72:0/3233145617 >> 172.21.15.72:0/3233145617 conn(0x7fa45c077610 msgr2=0x7fa45c103710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:37:35.309 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:35.307+0000 7fa46189d640 1 -- 172.21.15.72:0/3233145617 shutdown_connections 2024-03-26T02:37:35.309 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:37:35.307+0000 7fa46189d640 1 -- 172.21.15.72:0/3233145617 wait complete. 2024-03-26T02:37:35.320 INFO:tasks.workunit.client.0.smithi072.stderr:+ sleep 30 2024-03-26T02:37:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:35 smithi173 ceph-mon[33839]: pgmap v1844: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:37:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:35 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]': finished 2024-03-26T02:37:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:35 smithi173 ceph-mon[33839]: osdmap e841: 8 total, 8 up, 8 in 2024-03-26T02:37:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:35 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3233145617' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T02:37:35.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:35 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T02:37:35.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:35 smithi072 ceph-mon[41619]: pgmap v1844: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:37:35.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:35 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]': finished 2024-03-26T02:37:35.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:35 smithi072 ceph-mon[41619]: osdmap e841: 8 total, 8 up, 8 in 2024-03-26T02:37:35.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:35 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3233145617' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T02:37:35.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:35 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T02:37:35.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:35 smithi072 ceph-mon[30177]: pgmap v1844: 188 pgs: 188 active+clean; 859 KiB data, 900 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T02:37:35.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:35 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]': finished 2024-03-26T02:37:35.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:35 smithi072 ceph-mon[30177]: osdmap e841: 8 total, 8 up, 8 in 2024-03-26T02:37:35.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:35 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3233145617' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T02:37:35.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:35 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T02:37:36.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:36 smithi173 ceph-mon[33839]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]': finished 2024-03-26T02:37:36.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:36 smithi173 ceph-mon[33839]: osdmap e842: 8 total, 8 up, 8 in 2024-03-26T02:37:36.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:36 smithi072 ceph-mon[30177]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]': finished 2024-03-26T02:37:36.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:36 smithi072 ceph-mon[30177]: osdmap e842: 8 total, 8 up, 8 in 2024-03-26T02:37:36.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:36 smithi072 ceph-mon[41619]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "field": "max_objects", "val": "0"}]': finished 2024-03-26T02:37:36.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:36 smithi072 ceph-mon[41619]: osdmap e842: 8 total, 8 up, 8 in 2024-03-26T02:37:37.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:37 smithi173 ceph-mon[33839]: pgmap v1847: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-26T02:37:37.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:37 smithi072 ceph-mon[30177]: pgmap v1847: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-26T02:37:37.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:37 smithi072 ceph-mon[41619]: pgmap v1847: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-26T02:37:39.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:39 smithi072 ceph-mon[30177]: pgmap v1848: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 862 B/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-26T02:37:39.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:39 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:37:39.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:39 smithi072 ceph-mon[41619]: pgmap v1848: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 862 B/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-26T02:37:39.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:39 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:37:39.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:39 smithi173 ceph-mon[33839]: pgmap v1848: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 862 B/s rd, 2.0 KiB/s wr, 1 op/s 2024-03-26T02:37:39.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:39 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:37:41.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:41 smithi072 ceph-mon[30177]: pgmap v1849: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 739 B/s rd, 1.7 KiB/s wr, 0 op/s 2024-03-26T02:37:41.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:41 smithi072 ceph-mon[41619]: pgmap v1849: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 739 B/s rd, 1.7 KiB/s wr, 0 op/s 2024-03-26T02:37:41.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:41 smithi173 ceph-mon[33839]: pgmap v1849: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 739 B/s rd, 1.7 KiB/s wr, 0 op/s 2024-03-26T02:37:42.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:37:42 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:37:43.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:43 smithi072 ceph-mon[30177]: pgmap v1850: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-26T02:37:43.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:43 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:43.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:43 smithi072 ceph-mon[41619]: pgmap v1850: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-26T02:37:43.720 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:43 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:43.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:43 smithi173 ceph-mon[33839]: pgmap v1850: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-26T02:37:43.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:43 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:45.326 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:37:44 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:37:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:37:45.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:45 smithi072 ceph-mon[30177]: pgmap v1851: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-26T02:37:45.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:45 smithi072 ceph-mon[41619]: pgmap v1851: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-26T02:37:45.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:45 smithi173 ceph-mon[33839]: pgmap v1851: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-26T02:37:47.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:47 smithi072 ceph-mon[30177]: pgmap v1852: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.1 KiB/s wr, 1 op/s 2024-03-26T02:37:47.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:47 smithi072 ceph-mon[41619]: pgmap v1852: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.1 KiB/s wr, 1 op/s 2024-03-26T02:37:47.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:47 smithi173 ceph-mon[33839]: pgmap v1852: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.1 KiB/s wr, 1 op/s 2024-03-26T02:37:49.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:49 smithi072 ceph-mon[30177]: pgmap v1853: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:49.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:49 smithi072 ceph-mon[41619]: pgmap v1853: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:49.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:49 smithi173 ceph-mon[33839]: pgmap v1853: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:51.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:51 smithi072 ceph-mon[30177]: pgmap v1854: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:51.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:51 smithi072 ceph-mon[41619]: pgmap v1854: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:51.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:51 smithi173 ceph-mon[33839]: pgmap v1854: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:52.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:37:52 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:37:53.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:53 smithi072 ceph-mon[41619]: pgmap v1855: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:53.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:53 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:53.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:53 smithi072 ceph-mon[30177]: pgmap v1855: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:53.720 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:53 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:53.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:53 smithi173 ceph-mon[33839]: pgmap v1855: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:53.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:53 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:37:54.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:54 smithi072 ceph-mon[30177]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:37:54.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:54 smithi072 ceph-mon[41619]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:37:54.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:54 smithi173 ceph-mon[33839]: from='mgr.24467 172.21.15.72:0/940397005' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T02:37:55.348 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:37:54 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:37:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:37:55.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:55 smithi072 ceph-mon[30177]: pgmap v1856: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:55.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:55 smithi072 ceph-mon[41619]: pgmap v1856: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:55.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:55 smithi173 ceph-mon[33839]: pgmap v1856: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:57.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:57 smithi072 ceph-mon[30177]: pgmap v1857: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:57.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:57 smithi072 ceph-mon[41619]: pgmap v1857: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:57.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:57 smithi173 ceph-mon[33839]: pgmap v1857: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:37:59.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:37:59 smithi072 ceph-mon[30177]: pgmap v1858: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:59.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:37:59 smithi072 ceph-mon[41619]: pgmap v1858: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:37:59.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:37:59 smithi173 ceph-mon[33839]: pgmap v1858: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:01.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:38:01 smithi072 ceph-mon[30177]: pgmap v1859: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:01.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:38:01 smithi072 ceph-mon[41619]: pgmap v1859: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:01.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:38:01 smithi173 ceph-mon[33839]: pgmap v1859: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:02.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:38:02 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:38:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:38:03 smithi072 ceph-mon[30177]: pgmap v1860: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:03.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:38:03 smithi072 ceph-mon[30177]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:38:03.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:38:03 smithi072 ceph-mon[41619]: pgmap v1860: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:03.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:38:03 smithi072 ceph-mon[41619]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:38:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:38:03 smithi173 ceph-mon[33839]: pgmap v1860: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:03.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:38:03 smithi173 ceph-mon[33839]: from='client.24406 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T02:38:05.322 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph osd pool delete de5f38de-babb-481b-9bb5-ced33138378b de5f38de-babb-481b-9bb5-ced33138378b --yes-i-really-really-mean-it 2024-03-26T02:38:05.364 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:38:04 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: ::ffff:172.21.15.173 - - [26/Mar/2024:02:38:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T02:38:05.417 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.413+0000 7f7458ac3640 1 -- 172.21.15.72:0/1608264634 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7454102800 msgr2=0x7f7454102be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:38:05.417 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.413+0000 7f7458ac3640 1 --2- 172.21.15.72:0/1608264634 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7454102800 0x7f7454102be0 secure :-1 s=READY pgs=2590 cs=0 l=1 rev1=1 crypto rx=0x7f7444009860 tx=0x7f744402bc20 comp rx=0 tx=0).stop 2024-03-26T02:38:05.417 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.413+0000 7f7458ac3640 1 -- 172.21.15.72:0/1608264634 shutdown_connections 2024-03-26T02:38:05.417 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.413+0000 7f7458ac3640 1 --2- 172.21.15.72:0/1608264634 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f745410bbd0 0x7f745410e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:38:05.417 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.413+0000 7f7458ac3640 1 --2- 172.21.15.72:0/1608264634 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7454103120 0x7f745410b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:38:05.417 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.413+0000 7f7458ac3640 1 --2- 172.21.15.72:0/1608264634 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f7454102800 0x7f7454102be0 unknown :-1 s=CLOSED pgs=2590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:38:05.417 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.413+0000 7f7458ac3640 1 -- 172.21.15.72:0/1608264634 >> 172.21.15.72:0/1608264634 conn(0x7f7454077670 msgr2=0x7f7454077a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:38:05.417 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.413+0000 7f7458ac3640 1 -- 172.21.15.72:0/1608264634 shutdown_connections 2024-03-26T02:38:05.418 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.413+0000 7f7458ac3640 1 -- 172.21.15.72:0/1608264634 wait complete. 2024-03-26T02:38:05.418 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.414+0000 7f7458ac3640 1 Processor -- start 2024-03-26T02:38:05.418 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.414+0000 7f7458ac3640 1 -- start start 2024-03-26T02:38:05.418 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.414+0000 7f7458ac3640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7454102800 0x7f7454197340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:38:05.418 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.414+0000 7f7458ac3640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7454103120 0x7f7454197880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:38:05.418 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.414+0000 7f7458ac3640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f745410bbd0 0x7f745419bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:38:05.418 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.414+0000 7f7458ac3640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745419c1b0 con 0x7f7454102800 2024-03-26T02:38:05.418 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.414+0000 7f7458ac3640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745419c320 con 0x7f7454103120 2024-03-26T02:38:05.418 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.414+0000 7f7458ac3640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f745419c4e0 con 0x7f745410bbd0 2024-03-26T02:38:05.418 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.414+0000 7f745259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7454102800 0x7f7454197340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:38:05.418 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.415+0000 7f7452d9d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f745410bbd0 0x7f745419bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:38:05.418 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.415+0000 7f745259c640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7454102800 0x7f7454197340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40660/0 (socket says 172.21.15.72:40660) 2024-03-26T02:38:05.418 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.415+0000 7f745259c640 1 -- 172.21.15.72:0/3771413082 learned_addr learned my addr 172.21.15.72:0/3771413082 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:38:05.419 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.415+0000 7f744bfff640 1 --2- 172.21.15.72:0/3771413082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7454103120 0x7f7454197880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:38:05.419 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.415+0000 7f745259c640 1 -- 172.21.15.72:0/3771413082 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f745410bbd0 msgr2=0x7f745419bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:38:05.419 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.415+0000 7f745259c640 1 --2- 172.21.15.72:0/3771413082 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f745410bbd0 0x7f745419bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:38:05.419 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.415+0000 7f745259c640 1 -- 172.21.15.72:0/3771413082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7454103120 msgr2=0x7f7454197880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:38:05.419 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.415+0000 7f745259c640 1 --2- 172.21.15.72:0/3771413082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7454103120 0x7f7454197880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:38:05.419 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.415+0000 7f745259c640 1 -- 172.21.15.72:0/3771413082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f744c009900 con 0x7f7454102800 2024-03-26T02:38:05.419 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.415+0000 7f7452d9d640 1 --2- 172.21.15.72:0/3771413082 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f745410bbd0 0x7f745419bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-26T02:38:05.419 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.415+0000 7f744bfff640 1 --2- 172.21.15.72:0/3771413082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7454103120 0x7f7454197880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-26T02:38:05.419 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.415+0000 7f745259c640 1 --2- 172.21.15.72:0/3771413082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7454102800 0x7f7454197340 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f7444037040 tx=0x7f744402c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:38:05.419 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.415+0000 7f744b7fe640 1 -- 172.21.15.72:0/3771413082 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744402cc70 con 0x7f7454102800 2024-03-26T02:38:05.419 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.415+0000 7f7458ac3640 1 -- 172.21.15.72:0/3771413082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7444009490 con 0x7f7454102800 2024-03-26T02:38:05.419 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.415+0000 7f744b7fe640 1 -- 172.21.15.72:0/3771413082 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7f7444034af0 con 0x7f7454102800 2024-03-26T02:38:05.419 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.415+0000 7f744b7fe640 1 -- 172.21.15.72:0/3771413082 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f744403d3f0 con 0x7f7454102800 2024-03-26T02:38:05.420 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.415+0000 7f7458ac3640 1 -- 172.21.15.72:0/3771413082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74541a3e60 con 0x7f7454102800 2024-03-26T02:38:05.420 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.416+0000 7f7458ac3640 1 -- 172.21.15.72:0/3771413082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74540688c0 con 0x7f7454102800 2024-03-26T02:38:05.420 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.417+0000 7f744b7fe640 1 -- 172.21.15.72:0/3771413082 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7f744403b040 con 0x7f7454102800 2024-03-26T02:38:05.421 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.417+0000 7f744b7fe640 1 --2- 172.21.15.72:0/3771413082 >> v2:172.21.15.72:6800/3246690770 conn(0x7f744006d9f0 0x7f744006feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:38:05.421 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.417+0000 7f744b7fe640 1 -- 172.21.15.72:0/3771413082 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(842..842 src has 255..842) v4 ==== 8234+0+0 (secure 0 0 0) 0x7f74440af630 con 0x7f7454102800 2024-03-26T02:38:05.421 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.417+0000 7f744bfff640 1 --2- 172.21.15.72:0/3771413082 >> v2:172.21.15.72:6800/3246690770 conn(0x7f744006d9f0 0x7f744006feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:38:05.421 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.417+0000 7f744b7fe640 1 -- 172.21.15.72:0/3771413082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=843}) v3 -- 0x7f7440079700 con 0x7f7454102800 2024-03-26T02:38:05.421 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.418+0000 7f744bfff640 1 --2- 172.21.15.72:0/3771413082 >> v2:172.21.15.72:6800/3246690770 conn(0x7f744006d9f0 0x7f744006feb0 secure :-1 s=READY pgs=3405 cs=0 l=1 rev1=1 crypto rx=0x7f7454077e20 tx=0x7f743c005e50 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:38:05.422 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.422+0000 7f744b7fe640 1 -- 172.21.15.72:0/3771413082 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7f744407c300 con 0x7f7454102800 2024-03-26T02:38:05.598 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:05.597+0000 7f7458ac3640 1 -- 172.21.15.72:0/3771413082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pool2": "de5f38de-babb-481b-9bb5-ced33138378b", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f745419c6a0 con 0x7f7454102800 2024-03-26T02:38:05.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:38:05 smithi072 ceph-mon[30177]: pgmap v1861: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:05.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:38:05 smithi072 ceph-mon[41619]: pgmap v1861: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:05.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:38:05 smithi173 ceph-mon[33839]: pgmap v1861: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T02:38:06.369 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.367+0000 7f744b7fe640 1 -- 172.21.15.72:0/3771413082 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pool2": "de5f38de-babb-481b-9bb5-ced33138378b", "yes_i_really_really_mean_it": true}]=0 pool 'de5f38de-babb-481b-9bb5-ced33138378b' removed v843) v1 ==== 248+0+0 (secure 0 0 0) 0x7f7444039100 con 0x7f7454102800 2024-03-26T02:38:06.369 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.367+0000 7f744b7fe640 1 -- 172.21.15.72:0/3771413082 <== mon.0 v2:172.21.15.72:3300/0 8 ==== osd_map(843..843 src has 255..843) v4 ==== 296+0+0 (secure 0 0 0) 0x7f74440805d0 con 0x7f7454102800 2024-03-26T02:38:06.369 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.368+0000 7f744b7fe640 1 -- 172.21.15.72:0/3771413082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=844}) v3 -- 0x7f744007a380 con 0x7f7454102800 2024-03-26T02:38:06.486 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.484+0000 7f7458ac3640 1 -- 172.21.15.72:0/3771413082 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pool2": "de5f38de-babb-481b-9bb5-ced33138378b", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f74541063d0 con 0x7f7454102800 2024-03-26T02:38:06.486 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.484+0000 7f744b7fe640 1 -- 172.21.15.72:0/3771413082 <== mon.0 v2:172.21.15.72:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pool2": "de5f38de-babb-481b-9bb5-ced33138378b", "yes_i_really_really_mean_it": true}]=0 pool 'de5f38de-babb-481b-9bb5-ced33138378b' does not exist v843) v1 ==== 255+0+0 (secure 0 0 0) 0x7f744404b020 con 0x7f7454102800 2024-03-26T02:38:06.486 INFO:tasks.workunit.client.0.smithi072.stderr:pool 'de5f38de-babb-481b-9bb5-ced33138378b' does not exist 2024-03-26T02:38:06.488 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.486+0000 7f7458ac3640 1 -- 172.21.15.72:0/3771413082 >> v2:172.21.15.72:6800/3246690770 conn(0x7f744006d9f0 msgr2=0x7f744006feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:38:06.489 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.486+0000 7f7458ac3640 1 --2- 172.21.15.72:0/3771413082 >> v2:172.21.15.72:6800/3246690770 conn(0x7f744006d9f0 0x7f744006feb0 secure :-1 s=READY pgs=3405 cs=0 l=1 rev1=1 crypto rx=0x7f7454077e20 tx=0x7f743c005e50 comp rx=0 tx=0).stop 2024-03-26T02:38:06.489 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.486+0000 7f7458ac3640 1 -- 172.21.15.72:0/3771413082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7454102800 msgr2=0x7f7454197340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:38:06.489 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.486+0000 7f7458ac3640 1 --2- 172.21.15.72:0/3771413082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7454102800 0x7f7454197340 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f7444037040 tx=0x7f744402c870 comp rx=0 tx=0).stop 2024-03-26T02:38:06.489 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.487+0000 7f7458ac3640 1 -- 172.21.15.72:0/3771413082 shutdown_connections 2024-03-26T02:38:06.489 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.487+0000 7f7458ac3640 1 --2- 172.21.15.72:0/3771413082 >> v2:172.21.15.72:6800/3246690770 conn(0x7f744006d9f0 0x7f744006feb0 unknown :-1 s=CLOSED pgs=3405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:38:06.489 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.487+0000 7f7458ac3640 1 --2- 172.21.15.72:0/3771413082 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7f745410bbd0 0x7f745419bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:38:06.489 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.487+0000 7f7458ac3640 1 --2- 172.21.15.72:0/3771413082 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7454103120 0x7f7454197880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:38:06.489 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.487+0000 7f7458ac3640 1 --2- 172.21.15.72:0/3771413082 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7f7454102800 0x7f7454197340 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:38:06.489 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.487+0000 7f7458ac3640 1 -- 172.21.15.72:0/3771413082 >> 172.21.15.72:0/3771413082 conn(0x7f7454077670 msgr2=0x7f74541035a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:38:06.489 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.487+0000 7f7458ac3640 1 -- 172.21.15.72:0/3771413082 shutdown_connections 2024-03-26T02:38:06.489 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.487+0000 7f7458ac3640 1 -- 172.21.15.72:0/3771413082 wait complete. 2024-03-26T02:38:06.497 INFO:tasks.workunit.client.0.smithi072.stderr:+ ceph osd pool delete 6857fade-a9dd-4b5e-87a3-010f68d73cf2 6857fade-a9dd-4b5e-87a3-010f68d73cf2 --yes-i-really-really-mean-it 2024-03-26T02:38:06.581 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.577+0000 7fbfacced640 1 -- 172.21.15.72:0/4033746077 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfa8101f40 msgr2=0x7fbfa810e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:38:06.581 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.577+0000 7fbfacced640 1 --2- 172.21.15.72:0/4033746077 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfa8101f40 0x7fbfa810e400 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0009ca0 tx=0x7fbfa002c0f0 comp rx=0 tx=0).stop 2024-03-26T02:38:06.581 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.577+0000 7fbfacced640 1 -- 172.21.15.72:0/4033746077 shutdown_connections 2024-03-26T02:38:06.582 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.577+0000 7fbfacced640 1 --2- 172.21.15.72:0/4033746077 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfa8101f40 0x7fbfa810e400 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:38:06.582 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.577+0000 7fbfacced640 1 --2- 172.21.15.72:0/4033746077 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbfa8101580 0x7fbfa8101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:38:06.582 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.577+0000 7fbfacced640 1 --2- 172.21.15.72:0/4033746077 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbfa8107640 0x7fbfa8107a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:38:06.582 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.577+0000 7fbfacced640 1 -- 172.21.15.72:0/4033746077 >> 172.21.15.72:0/4033746077 conn(0x7fbfa80fd090 msgr2=0x7fbfa80ff4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:38:06.582 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.577+0000 7fbfacced640 1 -- 172.21.15.72:0/4033746077 shutdown_connections 2024-03-26T02:38:06.582 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.577+0000 7fbfacced640 1 -- 172.21.15.72:0/4033746077 wait complete. 2024-03-26T02:38:06.582 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.578+0000 7fbfacced640 1 Processor -- start 2024-03-26T02:38:06.582 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.578+0000 7fbfacced640 1 -- start start 2024-03-26T02:38:06.582 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.578+0000 7fbfacced640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbfa8101580 0x7fbfa8197260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:38:06.582 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.578+0000 7fbfacced640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfa8101f40 0x7fbfa81977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:38:06.582 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.578+0000 7fbfacced640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbfa8107640 0x7fbfa819bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:38:06.582 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.578+0000 7fbfacced640 1 -- --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa819c120 con 0x7fbfa8101f40 2024-03-26T02:38:06.583 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.578+0000 7fbfacced640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa819c290 con 0x7fbfa8101580 2024-03-26T02:38:06.583 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.578+0000 7fbfacced640 1 -- --> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa819c450 con 0x7fbfa8107640 2024-03-26T02:38:06.583 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.578+0000 7fbf9ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfa8101f40 0x7fbfa81977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:38:06.583 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.578+0000 7fbf9ffff640 1 --2- >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfa8101f40 0x7fbfa81977a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.72:3300/0 says I am v2:172.21.15.72:40684/0 (socket says 172.21.15.72:40684) 2024-03-26T02:38:06.583 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.578+0000 7fbfa6d9d640 1 --2- >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbfa8107640 0x7fbfa819bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:38:06.583 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.578+0000 7fbf9ffff640 1 -- 172.21.15.72:0/2308225288 learned_addr learned my addr 172.21.15.72:0/2308225288 (peer_addr_for_me v2:172.21.15.72:0/0) 2024-03-26T02:38:06.583 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.578+0000 7fbf9ffff640 1 -- 172.21.15.72:0/2308225288 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbfa8107640 msgr2=0x7fbfa819bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:38:06.583 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.578+0000 7fbf9ffff640 1 --2- 172.21.15.72:0/2308225288 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbfa8107640 0x7fbfa819bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:38:06.583 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.578+0000 7fbfa659c640 1 --2- 172.21.15.72:0/2308225288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbfa8101580 0x7fbfa8197260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:38:06.583 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.578+0000 7fbf9ffff640 1 -- 172.21.15.72:0/2308225288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbfa8101580 msgr2=0x7fbfa8197260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:38:06.583 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.578+0000 7fbf9ffff640 1 --2- 172.21.15.72:0/2308225288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbfa8101580 0x7fbfa8197260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:38:06.583 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.578+0000 7fbf9ffff640 1 -- 172.21.15.72:0/2308225288 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa0009900 con 0x7fbfa8101f40 2024-03-26T02:38:06.583 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.578+0000 7fbf9ffff640 1 --2- 172.21.15.72:0/2308225288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfa8101f40 0x7fbfa81977a0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7fbf9000b4a0 tx=0x7fbf9000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:38:06.583 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.579+0000 7fbf9f7fe640 1 -- 172.21.15.72:0/2308225288 <== mon.0 v2:172.21.15.72:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf90010610 con 0x7fbfa8101f40 2024-03-26T02:38:06.584 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.579+0000 7fbfacced640 1 -- 172.21.15.72:0/2308225288 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfa819c730 con 0x7fbfa8101f40 2024-03-26T02:38:06.584 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.579+0000 7fbf9f7fe640 1 -- 172.21.15.72:0/2308225288 <== mon.0 v2:172.21.15.72:3300/0 2 ==== config(26 keys) v1 ==== 1093+0+0 (secure 0 0 0) 0x7fbf90010c20 con 0x7fbfa8101f40 2024-03-26T02:38:06.584 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.579+0000 7fbf9f7fe640 1 -- 172.21.15.72:0/2308225288 <== mon.0 v2:172.21.15.72:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf90017cd0 con 0x7fbfa8101f40 2024-03-26T02:38:06.584 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.579+0000 7fbfacced640 1 -- 172.21.15.72:0/2308225288 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfa81a3fb0 con 0x7fbfa8101f40 2024-03-26T02:38:06.584 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.580+0000 7fbfacced640 1 -- 172.21.15.72:0/2308225288 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfa80688c0 con 0x7fbfa8101f40 2024-03-26T02:38:06.585 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.580+0000 7fbf9f7fe640 1 -- 172.21.15.72:0/2308225288 <== mon.0 v2:172.21.15.72:3300/0 4 ==== mgrmap(e 22) v1 ==== 91420+0+0 (secure 0 0 0) 0x7fbf90010770 con 0x7fbfa8101f40 2024-03-26T02:38:06.585 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.580+0000 7fbf9f7fe640 1 --2- 172.21.15.72:0/2308225288 >> v2:172.21.15.72:6800/3246690770 conn(0x7fbf9406d9f0 0x7fbf9406feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-26T02:38:06.585 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.581+0000 7fbf9f7fe640 1 -- 172.21.15.72:0/2308225288 <== mon.0 v2:172.21.15.72:3300/0 5 ==== osd_map(843..843 src has 255..843) v4 ==== 7846+0+0 (secure 0 0 0) 0x7fbf9008d060 con 0x7fbfa8101f40 2024-03-26T02:38:06.585 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.581+0000 7fbfa659c640 1 --2- 172.21.15.72:0/2308225288 >> v2:172.21.15.72:6800/3246690770 conn(0x7fbf9406d9f0 0x7fbf9406feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-26T02:38:06.585 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.581+0000 7fbf9f7fe640 1 -- 172.21.15.72:0/2308225288 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=844}) v3 -- 0x7fbf94078f60 con 0x7fbfa8101f40 2024-03-26T02:38:06.585 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.581+0000 7fbfa659c640 1 --2- 172.21.15.72:0/2308225288 >> v2:172.21.15.72:6800/3246690770 conn(0x7fbf9406d9f0 0x7fbf9406feb0 secure :-1 s=READY pgs=3406 cs=0 l=1 rev1=1 crypto rx=0x7fbf98009aa0 tx=0x7fbf98005f90 comp rx=0 tx=0).ready entity=mgr.24467 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-26T02:38:06.587 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.585+0000 7fbf9f7fe640 1 -- 172.21.15.72:0/2308225288 <== mon.0 v2:172.21.15.72:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (secure 0 0 0) 0x7fbf90059f00 con 0x7fbfa8101f40 2024-03-26T02:38:06.719 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:38:06 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3771413082' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pool2": "de5f38de-babb-481b-9bb5-ced33138378b", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T02:38:06.719 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:38:06 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3771413082' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pool2": "de5f38de-babb-481b-9bb5-ced33138378b", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T02:38:06.762 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:06.760+0000 7fbfacced640 1 -- 172.21.15.72:0/2308225288 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "pool2": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbfa8110e10 con 0x7fbfa8101f40 2024-03-26T02:38:06.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:38:06 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3771413082' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pool2": "de5f38de-babb-481b-9bb5-ced33138378b", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T02:38:07.375 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:07.373+0000 7fbf9f7fe640 1 -- 172.21.15.72:0/2308225288 <== mon.0 v2:172.21.15.72:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "pool2": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "yes_i_really_really_mean_it": true}]=0 pool '6857fade-a9dd-4b5e-87a3-010f68d73cf2' removed v844) v1 ==== 248+0+0 (secure 0 0 0) 0x7fbf90014100 con 0x7fbfa8101f40 2024-03-26T02:38:07.375 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:07.374+0000 7fbf9f7fe640 1 -- 172.21.15.72:0/2308225288 <== mon.0 v2:172.21.15.72:3300/0 8 ==== osd_map(844..844 src has 255..844) v4 ==== 296+0+0 (secure 0 0 0) 0x7fbf90017360 con 0x7fbfa8101f40 2024-03-26T02:38:07.375 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:07.374+0000 7fbf9f7fe640 1 -- 172.21.15.72:0/2308225288 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_subscribe({osdmap=845}) v3 -- 0x7fbf94079c00 con 0x7fbfa8101f40 2024-03-26T02:38:07.491 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:07.489+0000 7fbfacced640 1 -- 172.21.15.72:0/2308225288 --> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "pool2": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbfa81985b0 con 0x7fbfa8101f40 2024-03-26T02:38:07.491 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:07.489+0000 7fbf9f7fe640 1 -- 172.21.15.72:0/2308225288 <== mon.0 v2:172.21.15.72:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "pool2": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "yes_i_really_really_mean_it": true}]=0 pool '6857fade-a9dd-4b5e-87a3-010f68d73cf2' does not exist v844) v1 ==== 255+0+0 (secure 0 0 0) 0x7fbf9001e030 con 0x7fbfa8101f40 2024-03-26T02:38:07.491 INFO:tasks.workunit.client.0.smithi072.stderr:pool '6857fade-a9dd-4b5e-87a3-010f68d73cf2' does not exist 2024-03-26T02:38:07.493 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:07.491+0000 7fbfacced640 1 -- 172.21.15.72:0/2308225288 >> v2:172.21.15.72:6800/3246690770 conn(0x7fbf9406d9f0 msgr2=0x7fbf9406feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:38:07.493 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:07.491+0000 7fbfacced640 1 --2- 172.21.15.72:0/2308225288 >> v2:172.21.15.72:6800/3246690770 conn(0x7fbf9406d9f0 0x7fbf9406feb0 secure :-1 s=READY pgs=3406 cs=0 l=1 rev1=1 crypto rx=0x7fbf98009aa0 tx=0x7fbf98005f90 comp rx=0 tx=0).stop 2024-03-26T02:38:07.493 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:07.491+0000 7fbfacced640 1 -- 172.21.15.72:0/2308225288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfa8101f40 msgr2=0x7fbfa81977a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T02:38:07.494 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:07.491+0000 7fbfacced640 1 --2- 172.21.15.72:0/2308225288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfa8101f40 0x7fbfa81977a0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7fbf9000b4a0 tx=0x7fbf9000b960 comp rx=0 tx=0).stop 2024-03-26T02:38:07.494 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:07.491+0000 7fbfacced640 1 -- 172.21.15.72:0/2308225288 shutdown_connections 2024-03-26T02:38:07.494 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:07.491+0000 7fbfacced640 1 --2- 172.21.15.72:0/2308225288 >> v2:172.21.15.72:6800/3246690770 conn(0x7fbf9406d9f0 0x7fbf9406feb0 unknown :-1 s=CLOSED pgs=3406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:38:07.494 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:07.491+0000 7fbfacced640 1 --2- 172.21.15.72:0/2308225288 >> [v2:172.21.15.72:3301/0,v1:172.21.15.72:6790/0] conn(0x7fbfa8107640 0x7fbfa819bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:38:07.494 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:07.491+0000 7fbfacced640 1 --2- 172.21.15.72:0/2308225288 >> [v2:172.21.15.72:3300/0,v1:172.21.15.72:6789/0] conn(0x7fbfa8101f40 0x7fbfa81977a0 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:38:07.494 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:07.491+0000 7fbfacced640 1 --2- 172.21.15.72:0/2308225288 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbfa8101580 0x7fbfa8197260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-26T02:38:07.494 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:07.491+0000 7fbfacced640 1 -- 172.21.15.72:0/2308225288 >> 172.21.15.72:0/2308225288 conn(0x7fbfa80fd090 msgr2=0x7fbfa8105130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T02:38:07.494 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:07.491+0000 7fbfacced640 1 -- 172.21.15.72:0/2308225288 shutdown_connections 2024-03-26T02:38:07.494 INFO:tasks.workunit.client.0.smithi072.stderr:2024-03-26T02:38:07.491+0000 7fbfacced640 1 -- 172.21.15.72:0/2308225288 wait complete. 2024-03-26T02:38:07.502 INFO:tasks.workunit.client.0.smithi072.stderr:+ echo OK 2024-03-26T02:38:07.503 INFO:tasks.workunit.client.0.smithi072.stdout:OK 2024-03-26T02:38:07.503 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-26T02:38:07.504 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-26T02:38:07.573 INFO:tasks.workunit:Stopping ['rados/test.sh', 'rados/test_pool_quota.sh'] on client.0... 2024-03-26T02:38:07.573 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-26T02:38:07.637 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:38:07 smithi072 ceph-mon[30177]: pgmap v1862: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:07.637 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:38:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3771413082' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pool2": "de5f38de-babb-481b-9bb5-ced33138378b", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T02:38:07.638 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:38:07 smithi072 ceph-mon[30177]: osdmap e843: 8 total, 8 up, 8 in 2024-03-26T02:38:07.638 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:38:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/3771413082' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pool2": "de5f38de-babb-481b-9bb5-ced33138378b", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T02:38:07.638 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:38:07 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2308225288' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "pool2": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T02:38:07.638 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:38:07 smithi072 ceph-mon[41619]: pgmap v1862: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:07.638 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:38:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3771413082' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pool2": "de5f38de-babb-481b-9bb5-ced33138378b", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T02:38:07.638 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:38:07 smithi072 ceph-mon[41619]: osdmap e843: 8 total, 8 up, 8 in 2024-03-26T02:38:07.638 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:38:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/3771413082' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pool2": "de5f38de-babb-481b-9bb5-ced33138378b", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T02:38:07.638 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:38:07 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2308225288' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "pool2": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T02:38:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:38:07 smithi173 ceph-mon[33839]: pgmap v1862: 188 pgs: 188 active+clean; 871 KiB data, 918 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T02:38:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:38:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3771413082' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pool2": "de5f38de-babb-481b-9bb5-ced33138378b", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T02:38:07.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:38:07 smithi173 ceph-mon[33839]: osdmap e843: 8 total, 8 up, 8 in 2024-03-26T02:38:07.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:38:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/3771413082' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "de5f38de-babb-481b-9bb5-ced33138378b", "pool2": "de5f38de-babb-481b-9bb5-ced33138378b", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T02:38:07.814 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:38:07 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2308225288' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "pool2": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T02:38:07.895 DEBUG:teuthology.parallel:result is None 2024-03-26T02:38:07.896 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-26T02:38:07.962 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-26T02:38:07.962 DEBUG:teuthology.orchestra.run.smithi072:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-03-26T02:38:08.046 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-03-26T02:38:08.046 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-26T02:38:08.056 INFO:tasks.cephadm:Teardown begin 2024-03-26T02:38:08.057 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:38:08.113 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:38:08.149 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-26T02:38:08.149 DEBUG:teuthology.orchestra.run.smithi072:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T02:38:08.168 DEBUG:teuthology.orchestra.run.smithi173:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T02:38:08.206 INFO:tasks.cephadm:Stopping all daemons... 2024-03-26T02:38:08.206 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-26T02:38:08.206 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@mon.a 2024-03-26T02:38:08.505 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:38:08 smithi072 ceph-mon[41619]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T02:38:08.506 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:38:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2308225288' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "pool2": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T02:38:08.506 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:38:08 smithi072 ceph-mon[41619]: osdmap e844: 8 total, 8 up, 8 in 2024-03-26T02:38:08.506 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:38:08 smithi072 ceph-mon[41619]: from='client.? 172.21.15.72:0/2308225288' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "pool2": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T02:38:08.506 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:38:08 smithi072 systemd[1]: Stopping Ceph mon.a for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:38:08.506 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:38:08 smithi072 ceph-mon[30177]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T02:38:08.506 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:38:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2308225288' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "pool2": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T02:38:08.506 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:38:08 smithi072 ceph-mon[30177]: osdmap e844: 8 total, 8 up, 8 in 2024-03-26T02:38:08.506 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:38:08 smithi072 ceph-mon[30177]: from='client.? 172.21.15.72:0/2308225288' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "pool2": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T02:38:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:38:08 smithi173 ceph-mon[33839]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T02:38:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:38:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2308225288' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "pool2": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T02:38:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:38:08 smithi173 ceph-mon[33839]: osdmap e844: 8 total, 8 up, 8 in 2024-03-26T02:38:08.563 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:38:08 smithi173 ceph-mon[33839]: from='client.? 172.21.15.72:0/2308225288' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "pool2": "6857fade-a9dd-4b5e-87a3-010f68d73cf2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T02:38:08.807 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:38:08 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-a[30154]: 2024-03-26T02:38:08.503+0000 7fed4d6f0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:38:08.807 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:38:08 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-a[30154]: 2024-03-26T02:38:08.503+0000 7fed4d6f0700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-26T02:38:08.807 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:38:08 smithi072 podman[160390]: 2024-03-26 02:38:08.53592117 +0000 UTC m=+0.144362612 container died 52553733e232008acdda40c51484e26e7561b37ee6b2680a5a5090858c8021d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:38:08.807 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:38:08 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:38:08] ENGINE Bus STOPPING 2024-03-26T02:38:08.807 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:38:08 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:38:08] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T02:38:08.807 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:38:08 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:38:08] ENGINE Bus STOPPED 2024-03-26T02:38:08.808 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:38:08 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:38:08] ENGINE Bus STARTING 2024-03-26T02:38:08.808 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:38:08 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:38:08] ENGINE Serving on http://:::9283 2024-03-26T02:38:08.808 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:38:08 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:38:08] ENGINE Bus STARTED 2024-03-26T02:38:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:38:08 smithi072 podman[160390]: 2024-03-26 02:38:08.990617829 +0000 UTC m=+0.599059322 container cleanup 52553733e232008acdda40c51484e26e7561b37ee6b2680a5a5090858c8021d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-26T02:38:09.219 INFO:journalctl@ceph.mon.a.smithi072.stdout:Mar 26 02:38:08 smithi072 bash[160390]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-a 2024-03-26T02:38:09.408 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@mon.a.service' 2024-03-26T02:38:09.448 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:38:09.448 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-26T02:38:09.448 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-26T02:38:09.448 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@mon.c 2024-03-26T02:38:09.795 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:38:09 smithi072 systemd[1]: Stopping Ceph mon.c for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:38:09.795 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:38:09 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-c[41596]: 2024-03-26T02:38:09.772+0000 7f7d8d72a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:38:09.795 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:38:09 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-c[41596]: 2024-03-26T02:38:09.772+0000 7f7d8d72a700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-26T02:38:09.795 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:38:09 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:38:09] ENGINE Bus STOPPING 2024-03-26T02:38:09.795 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:38:09 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:38:09] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T02:38:09.795 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:38:09 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:38:09] ENGINE Bus STOPPED 2024-03-26T02:38:10.219 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:38:09 smithi072 podman[160533]: 2024-03-26 02:38:09.979120677 +0000 UTC m=+0.305474710 container died b7a640456d42468b40baf16dd7f8bfbda93725d767c65616dc70f97e2a0b9251 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-c, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-26T02:38:10.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:38:09 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:38:09] ENGINE Bus STARTING 2024-03-26T02:38:10.219 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:38:09 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:38:09] ENGINE Serving on http://:::9283 2024-03-26T02:38:10.220 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:38:09 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: [26/Mar/2024:02:38:09] ENGINE Bus STARTED 2024-03-26T02:38:10.595 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:38:10 smithi072 podman[160533]: 2024-03-26 02:38:10.325962674 +0000 UTC m=+0.652316701 container cleanup b7a640456d42468b40baf16dd7f8bfbda93725d767c65616dc70f97e2a0b9251 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-c, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-26T02:38:10.595 INFO:journalctl@ceph.mon.c.smithi072.stdout:Mar 26 02:38:10 smithi072 bash[160533]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-c 2024-03-26T02:38:10.738 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@mon.c.service' 2024-03-26T02:38:10.775 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:38:10.775 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-26T02:38:10.776 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-26T02:38:10.776 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@mon.b 2024-03-26T02:38:11.063 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:38:10 smithi173 systemd[1]: Stopping Ceph mon.b for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:38:11.500 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:38:11 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-b[33817]: 2024-03-26T02:38:11.193+0000 7f4561374700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:38:11.500 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:38:11 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-b[33817]: 2024-03-26T02:38:11.193+0000 7f4561374700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-26T02:38:11.500 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:38:11 smithi173 podman[79250]: 2024-03-26 02:38:11.224581075 +0000 UTC m=+0.248617293 container died 5d6891919918325caca9b0924f43b96649732458d28e4245bf2f555615ba61e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-26T02:38:11.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:38:11 smithi173 podman[79250]: 2024-03-26 02:38:11.665692516 +0000 UTC m=+0.689728744 container cleanup 5d6891919918325caca9b0924f43b96649732458d28e4245bf2f555615ba61e7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-26T02:38:11.813 INFO:journalctl@ceph.mon.b.smithi173.stdout:Mar 26 02:38:11 smithi173 bash[79250]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mon-b 2024-03-26T02:38:12.106 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@mon.b.service' 2024-03-26T02:38:12.152 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:38:12.152 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-26T02:38:12.152 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-26T02:38:12.153 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@mgr.y 2024-03-26T02:38:12.469 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:38:12 smithi072 systemd[1]: Stopping Ceph mgr.y for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:38:12.498 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:38:12 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug there is no tcmu-runner data available 2024-03-26T02:38:12.771 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:38:12 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y[30433]: 2024-03-26T02:38:12.470+0000 7f5d2b5eb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:38:12.771 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:38:12 smithi072 podman[160676]: 2024-03-26 02:38:12.502288262 +0000 UTC m=+0.157819774 container died fb1cfd7a9a912e8eb3a16927825a80cb19944ba977be10aec814ece8da829caa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-26T02:38:13.134 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:38:12 smithi072 podman[160676]: 2024-03-26 02:38:12.853793254 +0000 UTC m=+0.509324772 container cleanup fb1cfd7a9a912e8eb3a16927825a80cb19944ba977be10aec814ece8da829caa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-26T02:38:13.134 INFO:journalctl@ceph.mgr.y.smithi072.stdout:Mar 26 02:38:12 smithi072 bash[160676]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-y 2024-03-26T02:38:13.265 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@mgr.y.service' 2024-03-26T02:38:13.302 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:38:13.302 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-26T02:38:13.302 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-26T02:38:13.302 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@mgr.x 2024-03-26T02:38:13.607 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:38:13 smithi173 systemd[1]: Stopping Ceph mgr.x for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:38:13.922 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:38:13 smithi173 podman[79386]: 2024-03-26 02:38:13.645821506 +0000 UTC m=+0.137138904 container died 5cdf7c3ec8fe6efe8fd0532504e3c0e951628548717808f05b8e7051495aab73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-26T02:38:14.313 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:38:14 smithi173 podman[79386]: 2024-03-26 02:38:14.01205238 +0000 UTC m=+0.503369771 container cleanup 5cdf7c3ec8fe6efe8fd0532504e3c0e951628548717808f05b8e7051495aab73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:38:14.313 INFO:journalctl@ceph.mgr.x.smithi173.stdout:Mar 26 02:38:14 smithi173 bash[79386]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-mgr-x 2024-03-26T02:38:14.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@mgr.x.service' 2024-03-26T02:38:14.500 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:38:14.500 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-26T02:38:14.501 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-26T02:38:14.501 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.0 2024-03-26T02:38:14.648 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 26 02:38:14 smithi072 systemd[1]: Stopping Ceph osd.0 for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:38:14.969 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 26 02:38:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-0[46061]: 2024-03-26T02:38:14.822+0000 7f1834afe700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:38:14.969 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 26 02:38:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-0[46061]: 2024-03-26T02:38:14.822+0000 7f1834afe700 -1 osd.0 844 *** Got signal Terminated *** 2024-03-26T02:38:14.969 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 26 02:38:14 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-0[46061]: 2024-03-26T02:38:14.822+0000 7f1834afe700 -1 osd.0 844 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:38:20.469 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 26 02:38:20 smithi072 podman[160811]: 2024-03-26 02:38:20.21823952 +0000 UTC m=+5.502213548 container died c1e7ac7ea8bd497033ca5a44238eebb5eadf80668973da517019a24fe4597798 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124) 2024-03-26T02:38:20.852 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 26 02:38:20 smithi072 podman[160811]: 2024-03-26 02:38:20.589884999 +0000 UTC m=+5.873859028 container cleanup c1e7ac7ea8bd497033ca5a44238eebb5eadf80668973da517019a24fe4597798 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-5e70124) 2024-03-26T02:38:20.853 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 26 02:38:20 smithi072 bash[160811]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-0 2024-03-26T02:38:21.206 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 26 02:38:20 smithi072 podman[160845]: 2024-03-26 02:38:20.852214168 +0000 UTC m=+0.633378929 container remove c1e7ac7ea8bd497033ca5a44238eebb5eadf80668973da517019a24fe4597798 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-26T02:38:21.469 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 26 02:38:21 smithi072 podman[160941]: 2024-03-26 02:38:21.109145167 +0000 UTC m=+0.017023437 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:38:21.469 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 26 02:38:21 smithi072 podman[160941]: 2024-03-26 02:38:21.212571624 +0000 UTC m=+0.120449933 container create 3309e1cc984209e302bf821fd24132e45c12e59bf4689ea7703885ce9ae377a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-26T02:38:21.469 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 26 02:38:21 smithi072 podman[160941]: 2024-03-26 02:38:21.450227199 +0000 UTC m=+0.358105466 container init 3309e1cc984209e302bf821fd24132e45c12e59bf4689ea7703885ce9ae377a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-26T02:38:21.469 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 26 02:38:21 smithi072 podman[160941]: 2024-03-26 02:38:21.455325539 +0000 UTC m=+0.363203809 container start 3309e1cc984209e302bf821fd24132e45c12e59bf4689ea7703885ce9ae377a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True) 2024-03-26T02:38:21.969 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 26 02:38:21 smithi072 podman[160941]: 2024-03-26 02:38:21.551958091 +0000 UTC m=+0.459836361 container attach 3309e1cc984209e302bf821fd24132e45c12e59bf4689ea7703885ce9ae377a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-0-deactivate, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-26T02:38:21.969 INFO:journalctl@ceph.osd.0.smithi072.stdout:Mar 26 02:38:21 smithi072 podman[160941]: 2024-03-26 02:38:21.753223528 +0000 UTC m=+0.661101796 container died 3309e1cc984209e302bf821fd24132e45c12e59bf4689ea7703885ce9ae377a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-26T02:38:22.380 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.0.service' 2024-03-26T02:38:22.420 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:38:22.420 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-26T02:38:22.421 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-26T02:38:22.421 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.1 2024-03-26T02:38:22.719 INFO:journalctl@ceph.osd.1.smithi072.stdout:Mar 26 02:38:22 smithi072 systemd[1]: Stopping Ceph osd.1 for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:38:23.219 INFO:journalctl@ceph.osd.1.smithi072.stdout:Mar 26 02:38:22 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-1[52314]: 2024-03-26T02:38:22.783+0000 7f39a727e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:38:23.219 INFO:journalctl@ceph.osd.1.smithi072.stdout:Mar 26 02:38:22 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-1[52314]: 2024-03-26T02:38:22.783+0000 7f39a727e700 -1 osd.1 844 *** Got signal Terminated *** 2024-03-26T02:38:23.219 INFO:journalctl@ceph.osd.1.smithi072.stdout:Mar 26 02:38:22 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-1[52314]: 2024-03-26T02:38:22.783+0000 7f39a727e700 -1 osd.1 844 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:38:26.064 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:38:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:38:25.618Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.72:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.72:8765: connect: connection refused" 2024-03-26T02:38:26.064 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:38:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:38:25.619Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.72:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.72:8765: connect: connection refused" 2024-03-26T02:38:26.064 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:38:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:38:25.619Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.72:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.72:8765: connect: connection refused" 2024-03-26T02:38:26.064 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:38:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:38:25.621Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.72:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.72:8765: connect: connection refused" 2024-03-26T02:38:28.451 INFO:journalctl@ceph.osd.1.smithi072.stdout:Mar 26 02:38:28 smithi072 podman[161071]: 2024-03-26 02:38:28.181385719 +0000 UTC m=+5.546108437 container died 951e73c65530d3cf4e1c610f866eb8f485eb2dc0dda5b398a969075df282b7ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-26T02:38:28.719 INFO:journalctl@ceph.osd.1.smithi072.stdout:Mar 26 02:38:28 smithi072 podman[161071]: 2024-03-26 02:38:28.542063771 +0000 UTC m=+5.906786487 container cleanup 951e73c65530d3cf4e1c610f866eb8f485eb2dc0dda5b398a969075df282b7ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-1, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-26T02:38:28.719 INFO:journalctl@ceph.osd.1.smithi072.stdout:Mar 26 02:38:28 smithi072 bash[161071]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-1 2024-03-26T02:38:29.148 INFO:journalctl@ceph.osd.1.smithi072.stdout:Mar 26 02:38:28 smithi072 podman[161096]: 2024-03-26 02:38:28.829828833 +0000 UTC m=+0.645303139 container remove 951e73c65530d3cf4e1c610f866eb8f485eb2dc0dda5b398a969075df282b7ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-1, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-5e70124, ceph=True) 2024-03-26T02:38:29.444 INFO:journalctl@ceph.osd.1.smithi072.stdout:Mar 26 02:38:29 smithi072 podman[161193]: 2024-03-26 02:38:29.148137233 +0000 UTC m=+0.106954412 container create a256ead185aba2b65f1f58a347898452c7d114c7bc868b417fb1e82b3f88abf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319) 2024-03-26T02:38:29.444 INFO:journalctl@ceph.osd.1.smithi072.stdout:Mar 26 02:38:29 smithi072 podman[161193]: 2024-03-26 02:38:29.059688828 +0000 UTC m=+0.018506015 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:38:29.444 INFO:journalctl@ceph.osd.1.smithi072.stdout:Mar 26 02:38:29 smithi072 podman[161193]: 2024-03-26 02:38:29.352968846 +0000 UTC m=+0.311786027 container init a256ead185aba2b65f1f58a347898452c7d114c7bc868b417fb1e82b3f88abf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=) 2024-03-26T02:38:29.444 INFO:journalctl@ceph.osd.1.smithi072.stdout:Mar 26 02:38:29 smithi072 podman[161193]: 2024-03-26 02:38:29.358648381 +0000 UTC m=+0.317465563 container start a256ead185aba2b65f1f58a347898452c7d114c7bc868b417fb1e82b3f88abf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-5e70124, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-26T02:38:29.444 INFO:journalctl@ceph.osd.1.smithi072.stdout:Mar 26 02:38:29 smithi072 podman[161193]: 2024-03-26 02:38:29.443921734 +0000 UTC m=+0.402738944 container attach a256ead185aba2b65f1f58a347898452c7d114c7bc868b417fb1e82b3f88abf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-1-deactivate, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-26T02:38:29.719 INFO:journalctl@ceph.osd.1.smithi072.stdout:Mar 26 02:38:29 smithi072 podman[161193]: 2024-03-26 02:38:29.645556041 +0000 UTC m=+0.604373219 container died a256ead185aba2b65f1f58a347898452c7d114c7bc868b417fb1e82b3f88abf5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-26T02:38:30.281 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.1.service' 2024-03-26T02:38:30.322 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:38:30.322 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-26T02:38:30.322 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-26T02:38:30.322 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.2 2024-03-26T02:38:30.651 INFO:journalctl@ceph.osd.2.smithi072.stdout:Mar 26 02:38:30 smithi072 systemd[1]: Stopping Ceph osd.2 for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:38:30.969 INFO:journalctl@ceph.osd.2.smithi072.stdout:Mar 26 02:38:30 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-2[58463]: 2024-03-26T02:38:30.650+0000 7f43eb46b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:38:30.969 INFO:journalctl@ceph.osd.2.smithi072.stdout:Mar 26 02:38:30 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-2[58463]: 2024-03-26T02:38:30.650+0000 7f43eb46b700 -1 osd.2 844 *** Got signal Terminated *** 2024-03-26T02:38:30.969 INFO:journalctl@ceph.osd.2.smithi072.stdout:Mar 26 02:38:30 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-2[58463]: 2024-03-26T02:38:30.650+0000 7f43eb46b700 -1 osd.2 844 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:38:36.327 INFO:journalctl@ceph.osd.2.smithi072.stdout:Mar 26 02:38:36 smithi072 podman[161326]: 2024-03-26 02:38:36.062127677 +0000 UTC m=+5.518889746 container died 8702565aa1c93fcf540026b0ec96bba06824ded19dfd879f4c19ce77ce1a6088 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-26T02:38:36.719 INFO:journalctl@ceph.osd.2.smithi072.stdout:Mar 26 02:38:36 smithi072 podman[161326]: 2024-03-26 02:38:36.426297346 +0000 UTC m=+5.883059404 container cleanup 8702565aa1c93fcf540026b0ec96bba06824ded19dfd879f4c19ce77ce1a6088 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-2, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=) 2024-03-26T02:38:36.719 INFO:journalctl@ceph.osd.2.smithi072.stdout:Mar 26 02:38:36 smithi072 bash[161326]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-2 2024-03-26T02:38:37.172 INFO:journalctl@ceph.osd.2.smithi072.stdout:Mar 26 02:38:36 smithi072 podman[161351]: 2024-03-26 02:38:36.82874336 +0000 UTC m=+0.761415678 container remove 8702565aa1c93fcf540026b0ec96bba06824ded19dfd879f4c19ce77ce1a6088 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-26T02:38:37.453 INFO:journalctl@ceph.osd.2.smithi072.stdout:Mar 26 02:38:37 smithi072 podman[161449]: 2024-03-26 02:38:37.172043241 +0000 UTC m=+0.105864751 container create f98e10e823c0fcd11cfbb48cb86776fbc7a70b2d7e7a27e4f4f4908acf867ae7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-2-deactivate, ceph=True, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-26T02:38:37.453 INFO:journalctl@ceph.osd.2.smithi072.stdout:Mar 26 02:38:37 smithi072 podman[161449]: 2024-03-26 02:38:37.084103441 +0000 UTC m=+0.017924964 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:38:37.454 INFO:journalctl@ceph.osd.2.smithi072.stdout:Mar 26 02:38:37 smithi072 podman[161449]: 2024-03-26 02:38:37.377558691 +0000 UTC m=+0.311380205 container init f98e10e823c0fcd11cfbb48cb86776fbc7a70b2d7e7a27e4f4f4908acf867ae7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-2-deactivate, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-26T02:38:37.454 INFO:journalctl@ceph.osd.2.smithi072.stdout:Mar 26 02:38:37 smithi072 podman[161449]: 2024-03-26 02:38:37.382713311 +0000 UTC m=+0.316534831 container start f98e10e823c0fcd11cfbb48cb86776fbc7a70b2d7e7a27e4f4f4908acf867ae7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.vendor=CentOS) 2024-03-26T02:38:37.719 INFO:journalctl@ceph.osd.2.smithi072.stdout:Mar 26 02:38:37 smithi072 podman[161449]: 2024-03-26 02:38:37.452642762 +0000 UTC m=+0.386464288 container attach f98e10e823c0fcd11cfbb48cb86776fbc7a70b2d7e7a27e4f4f4908acf867ae7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:38:37.719 INFO:journalctl@ceph.osd.2.smithi072.stdout:Mar 26 02:38:37 smithi072 podman[161449]: 2024-03-26 02:38:37.672800791 +0000 UTC m=+0.606622302 container died f98e10e823c0fcd11cfbb48cb86776fbc7a70b2d7e7a27e4f4f4908acf867ae7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-26T02:38:38.279 INFO:journalctl@ceph.osd.2.smithi072.stdout:Mar 26 02:38:38 smithi072 podman[161496]: 2024-03-26 02:38:38.252224734 +0000 UTC m=+0.574380364 container remove f98e10e823c0fcd11cfbb48cb86776fbc7a70b2d7e7a27e4f4f4908acf867ae7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:38:38.284 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.2.service' 2024-03-26T02:38:38.326 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:38:38.326 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-26T02:38:38.326 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-26T02:38:38.326 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.3 2024-03-26T02:38:38.667 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 26 02:38:38 smithi072 systemd[1]: Stopping Ceph osd.3 for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:38:38.969 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 26 02:38:38 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-3[65068]: 2024-03-26T02:38:38.665+0000 7fd1e46ea700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:38:38.969 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 26 02:38:38 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-3[65068]: 2024-03-26T02:38:38.665+0000 7fd1e46ea700 -1 osd.3 844 *** Got signal Terminated *** 2024-03-26T02:38:38.970 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 26 02:38:38 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-3[65068]: 2024-03-26T02:38:38.665+0000 7fd1e46ea700 -1 osd.3 844 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:38:41.313 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:40 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:40.849+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:15.868353+0000 front 2024-03-26T02:38:15.868227+0000 (oldest deadline 2024-03-26T02:38:40.568129+0000) 2024-03-26T02:38:42.215 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:41 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:41.885+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:15.868353+0000 front 2024-03-26T02:38:15.868227+0000 (oldest deadline 2024-03-26T02:38:40.568129+0000) 2024-03-26T02:38:43.163 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 02:38:43 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-4[39374]: 2024-03-26T02:38:43.006+0000 7fc2f3e1f700 -1 osd.4 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:16.912010+0000 front 2024-03-26T02:38:16.911874+0000 (oldest deadline 2024-03-26T02:38:42.211857+0000) 2024-03-26T02:38:43.164 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:42 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:42.908+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:15.868353+0000 front 2024-03-26T02:38:15.868227+0000 (oldest deadline 2024-03-26T02:38:40.568129+0000) 2024-03-26T02:38:43.431 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:43 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:43.162+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:38:44.064 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 02:38:44 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-4[39374]: 2024-03-26T02:38:44.023+0000 7fc2f3e1f700 -1 osd.4 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:16.912010+0000 front 2024-03-26T02:38:16.911874+0000 (oldest deadline 2024-03-26T02:38:42.211857+0000) 2024-03-26T02:38:44.064 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:43 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:43.951+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:15.868353+0000 front 2024-03-26T02:38:15.868227+0000 (oldest deadline 2024-03-26T02:38:40.568129+0000) 2024-03-26T02:38:44.064 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:43 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:43.790+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.726271+0000 front 2024-03-26T02:38:14.726289+0000 (oldest deadline 2024-03-26T02:38:43.526718+0000) 2024-03-26T02:38:44.333 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 26 02:38:44 smithi072 podman[161580]: 2024-03-26 02:38:44.080369852 +0000 UTC m=+5.535499392 container died 2895ed30f7434d1fda18fa711793ac1a7580338992b646ba38775e598e3ff4d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-3, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:38:44.563 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:44 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:44.166+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:38:44.704 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 26 02:38:44 smithi072 podman[161580]: 2024-03-26 02:38:44.442856684 +0000 UTC m=+5.897986212 container cleanup 2895ed30f7434d1fda18fa711793ac1a7580338992b646ba38775e598e3ff4d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-3, RELEASE=squid-5e70124, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-26T02:38:44.704 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 26 02:38:44 smithi072 bash[161580]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-3 2024-03-26T02:38:44.969 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 26 02:38:44 smithi072 podman[161605]: 2024-03-26 02:38:44.70358969 +0000 UTC m=+0.622781625 container remove 2895ed30f7434d1fda18fa711793ac1a7580338992b646ba38775e598e3ff4d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, ceph=True) 2024-03-26T02:38:45.057 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:44 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:44.973+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:15.868353+0000 front 2024-03-26T02:38:15.868227+0000 (oldest deadline 2024-03-26T02:38:40.568129+0000) 2024-03-26T02:38:45.057 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:44 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:44.764+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.726271+0000 front 2024-03-26T02:38:14.726289+0000 (oldest deadline 2024-03-26T02:38:43.526718+0000) 2024-03-26T02:38:45.313 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 02:38:45 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-4[39374]: 2024-03-26T02:38:45.056+0000 7fc2f3e1f700 -1 osd.4 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:16.912010+0000 front 2024-03-26T02:38:16.911874+0000 (oldest deadline 2024-03-26T02:38:42.211857+0000) 2024-03-26T02:38:45.313 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:45 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:45.138+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:38:45.361 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 26 02:38:45 smithi072 podman[161701]: 2024-03-26 02:38:44.958739277 +0000 UTC m=+0.017622580 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:38:45.361 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 26 02:38:45 smithi072 podman[161701]: 2024-03-26 02:38:45.071958486 +0000 UTC m=+0.130841776 container create 5f13f05f07f8644c1821d8280b917afa54b08989160315004ea5d8301bc75e18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, GIT_CLEAN=True) 2024-03-26T02:38:45.361 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 26 02:38:45 smithi072 podman[161701]: 2024-03-26 02:38:45.267392377 +0000 UTC m=+0.326275668 container init 5f13f05f07f8644c1821d8280b917afa54b08989160315004ea5d8301bc75e18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-26T02:38:45.361 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 26 02:38:45 smithi072 podman[161701]: 2024-03-26 02:38:45.273826943 +0000 UTC m=+0.332710234 container start 5f13f05f07f8644c1821d8280b917afa54b08989160315004ea5d8301bc75e18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-26T02:38:45.361 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 26 02:38:45 smithi072 podman[161701]: 2024-03-26 02:38:45.360804462 +0000 UTC m=+0.419687775 container attach 5f13f05f07f8644c1821d8280b917afa54b08989160315004ea5d8301bc75e18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-26T02:38:45.719 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 26 02:38:45 smithi072 podman[161747]: 2024-03-26 02:38:45.587486406 +0000 UTC m=+0.017966906 container died 5f13f05f07f8644c1821d8280b917afa54b08989160315004ea5d8301bc75e18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-26T02:38:46.063 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 02:38:46 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-4[39374]: 2024-03-26T02:38:46.032+0000 7fc2f3e1f700 -1 osd.4 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:16.912010+0000 front 2024-03-26T02:38:16.911874+0000 (oldest deadline 2024-03-26T02:38:42.211857+0000) 2024-03-26T02:38:46.063 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:45 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:45.966+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:15.868353+0000 front 2024-03-26T02:38:15.868227+0000 (oldest deadline 2024-03-26T02:38:40.568129+0000) 2024-03-26T02:38:46.064 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:45 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:45.807+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.726271+0000 front 2024-03-26T02:38:14.726289+0000 (oldest deadline 2024-03-26T02:38:43.526718+0000) 2024-03-26T02:38:46.087 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.3.service' 2024-03-26T02:38:46.114 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 26 02:38:46 smithi072 podman[161747]: 2024-03-26 02:38:46.067088166 +0000 UTC m=+0.497568705 container remove 5f13f05f07f8644c1821d8280b917afa54b08989160315004ea5d8301bc75e18 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-26T02:38:46.114 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 26 02:38:46 smithi072 systemd[1]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.3.service: Deactivated successfully. 2024-03-26T02:38:46.114 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 26 02:38:46 smithi072 systemd[1]: Stopped Ceph osd.3 for c86fbdd4-eb13-11ee-95d0-87774f69a715. 2024-03-26T02:38:46.114 INFO:journalctl@ceph.osd.3.smithi072.stdout:Mar 26 02:38:46 smithi072 systemd[1]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.3.service: Consumed 57.311s CPU time. 2024-03-26T02:38:46.130 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:38:46.130 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-26T02:38:46.130 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-26T02:38:46.130 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.4 2024-03-26T02:38:46.517 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 02:38:46 smithi173 systemd[1]: Stopping Ceph osd.4 for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:38:46.517 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:46 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:46.180+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:38:46.813 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 02:38:46 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-4[39374]: 2024-03-26T02:38:46.516+0000 7fc2f782b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:38:46.813 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 02:38:46 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-4[39374]: 2024-03-26T02:38:46.516+0000 7fc2f782b700 -1 osd.4 844 *** Got signal Terminated *** 2024-03-26T02:38:46.813 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 02:38:46 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-4[39374]: 2024-03-26T02:38:46.516+0000 7fc2f782b700 -1 osd.4 844 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:38:47.171 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 02:38:47 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-4[39374]: 2024-03-26T02:38:47.048+0000 7fc2f3e1f700 -1 osd.4 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:16.912010+0000 front 2024-03-26T02:38:16.911874+0000 (oldest deadline 2024-03-26T02:38:42.211857+0000) 2024-03-26T02:38:47.171 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:46 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:46.831+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.726271+0000 front 2024-03-26T02:38:14.726289+0000 (oldest deadline 2024-03-26T02:38:43.526718+0000) 2024-03-26T02:38:47.171 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:46 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:46.953+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:15.868353+0000 front 2024-03-26T02:38:15.868227+0000 (oldest deadline 2024-03-26T02:38:40.568129+0000) 2024-03-26T02:38:47.563 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:47 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:47.170+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:38:48.186 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:47 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:47.989+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:15.868353+0000 front 2024-03-26T02:38:15.868227+0000 (oldest deadline 2024-03-26T02:38:40.568129+0000) 2024-03-26T02:38:48.186 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:47 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:47.852+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.726271+0000 front 2024-03-26T02:38:14.726289+0000 (oldest deadline 2024-03-26T02:38:43.526718+0000) 2024-03-26T02:38:48.186 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 02:38:48 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-4[39374]: 2024-03-26T02:38:48.031+0000 7fc2f3e1f700 -1 osd.4 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:16.912010+0000 front 2024-03-26T02:38:16.911874+0000 (oldest deadline 2024-03-26T02:38:42.211857+0000) 2024-03-26T02:38:48.563 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:48 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:48.184+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:38:49.167 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 02:38:49 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-4[39374]: 2024-03-26T02:38:49.054+0000 7fc2f3e1f700 -1 osd.4 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:16.912010+0000 front 2024-03-26T02:38:16.911874+0000 (oldest deadline 2024-03-26T02:38:42.211857+0000) 2024-03-26T02:38:49.167 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:48 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:48.984+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:15.868353+0000 front 2024-03-26T02:38:15.868227+0000 (oldest deadline 2024-03-26T02:38:40.568129+0000) 2024-03-26T02:38:49.167 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:48 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:48.863+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.726271+0000 front 2024-03-26T02:38:14.726289+0000 (oldest deadline 2024-03-26T02:38:43.526718+0000) 2024-03-26T02:38:49.168 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:48 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:48.863+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:23.527317+0000 front 2024-03-26T02:38:23.527304+0000 (oldest deadline 2024-03-26T02:38:48.827213+0000) 2024-03-26T02:38:49.563 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:49 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:49.165+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:38:50.125 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:49 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:49.988+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:15.868353+0000 front 2024-03-26T02:38:15.868227+0000 (oldest deadline 2024-03-26T02:38:40.568129+0000) 2024-03-26T02:38:50.126 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 02:38:50 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-4[39374]: 2024-03-26T02:38:50.075+0000 7fc2f3e1f700 -1 osd.4 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:16.912010+0000 front 2024-03-26T02:38:16.911874+0000 (oldest deadline 2024-03-26T02:38:42.211857+0000) 2024-03-26T02:38:50.126 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 02:38:50 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-4[39374]: 2024-03-26T02:38:50.075+0000 7fc2f3e1f700 -1 osd.4 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:26.213303+0000 front 2024-03-26T02:38:26.213397+0000 (oldest deadline 2024-03-26T02:38:49.113281+0000) 2024-03-26T02:38:50.126 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:49 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:49.838+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.726271+0000 front 2024-03-26T02:38:14.726289+0000 (oldest deadline 2024-03-26T02:38:43.526718+0000) 2024-03-26T02:38:50.126 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:49 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:49.838+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:23.527317+0000 front 2024-03-26T02:38:23.527304+0000 (oldest deadline 2024-03-26T02:38:48.827213+0000) 2024-03-26T02:38:50.563 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:50 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:50.123+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:38:51.125 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:50 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:50.947+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:15.868353+0000 front 2024-03-26T02:38:15.868227+0000 (oldest deadline 2024-03-26T02:38:40.568129+0000) 2024-03-26T02:38:51.126 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 02:38:51 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-4[39374]: 2024-03-26T02:38:51.064+0000 7fc2f3e1f700 -1 osd.4 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:16.912010+0000 front 2024-03-26T02:38:16.911874+0000 (oldest deadline 2024-03-26T02:38:42.211857+0000) 2024-03-26T02:38:51.126 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 02:38:51 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-4[39374]: 2024-03-26T02:38:51.064+0000 7fc2f3e1f700 -1 osd.4 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:26.213303+0000 front 2024-03-26T02:38:26.213397+0000 (oldest deadline 2024-03-26T02:38:49.113281+0000) 2024-03-26T02:38:51.126 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:50 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:50.873+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.726271+0000 front 2024-03-26T02:38:14.726289+0000 (oldest deadline 2024-03-26T02:38:43.526718+0000) 2024-03-26T02:38:51.126 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:50 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:50.873+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:23.527317+0000 front 2024-03-26T02:38:23.527304+0000 (oldest deadline 2024-03-26T02:38:48.827213+0000) 2024-03-26T02:38:51.563 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:51 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:51.124+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:38:51.563 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:51 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:51.124+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:25.673886+0000 front 2024-03-26T02:38:25.673968+0000 (oldest deadline 2024-03-26T02:38:50.373763+0000) 2024-03-26T02:38:52.160 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 02:38:52 smithi173 podman[79544]: 2024-03-26 02:38:52.066116918 +0000 UTC m=+5.728453317 container died 81060cf483a5c3cdf84c54f57ff85c202fa07e1e68b16e330de9983c05ab678e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124) 2024-03-26T02:38:52.160 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:51 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:51.983+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:15.868353+0000 front 2024-03-26T02:38:15.868227+0000 (oldest deadline 2024-03-26T02:38:40.568129+0000) 2024-03-26T02:38:52.160 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:51 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:51.851+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.726271+0000 front 2024-03-26T02:38:14.726289+0000 (oldest deadline 2024-03-26T02:38:43.526718+0000) 2024-03-26T02:38:52.160 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:51 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:51.851+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:23.527317+0000 front 2024-03-26T02:38:23.527304+0000 (oldest deadline 2024-03-26T02:38:48.827213+0000) 2024-03-26T02:38:52.422 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:52 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:52.158+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:38:52.422 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:52 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:52.158+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:25.673886+0000 front 2024-03-26T02:38:25.673968+0000 (oldest deadline 2024-03-26T02:38:50.373763+0000) 2024-03-26T02:38:52.682 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 02:38:52 smithi173 podman[79544]: 2024-03-26 02:38:52.422212844 +0000 UTC m=+6.084549241 container cleanup 81060cf483a5c3cdf84c54f57ff85c202fa07e1e68b16e330de9983c05ab678e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-26T02:38:52.682 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 02:38:52 smithi173 bash[79544]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-4 2024-03-26T02:38:52.938 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 02:38:52 smithi173 podman[79567]: 2024-03-26 02:38:52.681853326 +0000 UTC m=+0.610248357 container remove 81060cf483a5c3cdf84c54f57ff85c202fa07e1e68b16e330de9983c05ab678e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-4, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-26T02:38:52.938 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:52 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:52.876+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.726271+0000 front 2024-03-26T02:38:14.726289+0000 (oldest deadline 2024-03-26T02:38:43.526718+0000) 2024-03-26T02:38:52.939 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:52 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:52.876+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:23.527317+0000 front 2024-03-26T02:38:23.527304+0000 (oldest deadline 2024-03-26T02:38:48.827213+0000) 2024-03-26T02:38:53.260 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 02:38:53 smithi173 podman[79662]: 2024-03-26 02:38:52.939315357 +0000 UTC m=+0.020636941 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:38:53.261 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 02:38:53 smithi173 podman[79662]: 2024-03-26 02:38:53.066368548 +0000 UTC m=+0.147690162 container create 218883490b88cc54df7ba14439103533098e442ace97e1f7d358a5d10e91fa62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:38:53.261 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:52 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:52.995+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:15.868353+0000 front 2024-03-26T02:38:15.868227+0000 (oldest deadline 2024-03-26T02:38:40.568129+0000) 2024-03-26T02:38:53.261 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:52 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:52.996+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:26.469074+0000 front 2024-03-26T02:38:26.469103+0000 (oldest deadline 2024-03-26T02:38:52.269363+0000) 2024-03-26T02:38:53.261 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:53 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:53.163+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:38:53.261 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:53 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:53.163+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:25.673886+0000 front 2024-03-26T02:38:25.673968+0000 (oldest deadline 2024-03-26T02:38:50.373763+0000) 2024-03-26T02:38:53.563 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 02:38:53 smithi173 podman[79662]: 2024-03-26 02:38:53.260371723 +0000 UTC m=+0.341693294 container init 218883490b88cc54df7ba14439103533098e442ace97e1f7d358a5d10e91fa62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, io.buildah.version=1.33.5, ceph=True) 2024-03-26T02:38:53.563 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 02:38:53 smithi173 podman[79662]: 2024-03-26 02:38:53.265745353 +0000 UTC m=+0.347066926 container start 218883490b88cc54df7ba14439103533098e442ace97e1f7d358a5d10e91fa62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-26T02:38:53.564 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 02:38:53 smithi173 podman[79662]: 2024-03-26 02:38:53.343471086 +0000 UTC m=+0.424792658 container attach 218883490b88cc54df7ba14439103533098e442ace97e1f7d358a5d10e91fa62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-4-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:38:53.829 INFO:journalctl@ceph.osd.4.smithi173.stdout:Mar 26 02:38:53 smithi173 podman[79662]: 2024-03-26 02:38:53.574761026 +0000 UTC m=+0.656082605 container died 218883490b88cc54df7ba14439103533098e442ace97e1f7d358a5d10e91fa62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T02:38:54.115 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:54 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:54.015+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:15.868353+0000 front 2024-03-26T02:38:15.868227+0000 (oldest deadline 2024-03-26T02:38:40.568129+0000) 2024-03-26T02:38:54.115 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:54 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:54.015+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:26.469074+0000 front 2024-03-26T02:38:26.469103+0000 (oldest deadline 2024-03-26T02:38:52.269363+0000) 2024-03-26T02:38:54.115 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:53 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:53.828+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.726271+0000 front 2024-03-26T02:38:14.726289+0000 (oldest deadline 2024-03-26T02:38:43.526718+0000) 2024-03-26T02:38:54.115 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:53 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:53.828+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:23.527317+0000 front 2024-03-26T02:38:23.527304+0000 (oldest deadline 2024-03-26T02:38:48.827213+0000) 2024-03-26T02:38:54.232 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.4.service' 2024-03-26T02:38:54.273 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:38:54.273 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-26T02:38:54.273 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-26T02:38:54.274 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.5 2024-03-26T02:38:54.563 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:54 smithi173 systemd[1]: Stopping Ceph osd.5 for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:38:54.564 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:54 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:54.113+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:38:54.564 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:54 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:54.113+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:25.673886+0000 front 2024-03-26T02:38:25.673968+0000 (oldest deadline 2024-03-26T02:38:50.373763+0000) 2024-03-26T02:38:55.006 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:54 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:54.825+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.726271+0000 front 2024-03-26T02:38:14.726289+0000 (oldest deadline 2024-03-26T02:38:43.526718+0000) 2024-03-26T02:38:55.007 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:54 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:54.825+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:23.527317+0000 front 2024-03-26T02:38:23.527304+0000 (oldest deadline 2024-03-26T02:38:48.827213+0000) 2024-03-26T02:38:55.007 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:54 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:54.603+0000 7f1d7575c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:38:55.007 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:54 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:54.603+0000 7f1d7575c700 -1 osd.5 844 *** Got signal Terminated *** 2024-03-26T02:38:55.007 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:54 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:54.603+0000 7f1d7575c700 -1 osd.5 844 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:38:55.007 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:55.005+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:15.868353+0000 front 2024-03-26T02:38:15.868227+0000 (oldest deadline 2024-03-26T02:38:40.568129+0000) 2024-03-26T02:38:55.007 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:55.005+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:26.469074+0000 front 2024-03-26T02:38:26.469103+0000 (oldest deadline 2024-03-26T02:38:52.269363+0000) 2024-03-26T02:38:55.313 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:55.069+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:38:55.313 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:55.069+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:25.673886+0000 front 2024-03-26T02:38:25.673968+0000 (oldest deadline 2024-03-26T02:38:50.373763+0000) 2024-03-26T02:38:56.043 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:55.781+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.726271+0000 front 2024-03-26T02:38:14.726289+0000 (oldest deadline 2024-03-26T02:38:43.526718+0000) 2024-03-26T02:38:56.043 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:55.781+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:23.527317+0000 front 2024-03-26T02:38:23.527304+0000 (oldest deadline 2024-03-26T02:38:48.827213+0000) 2024-03-26T02:38:56.043 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:55.987+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:15.868353+0000 front 2024-03-26T02:38:15.868227+0000 (oldest deadline 2024-03-26T02:38:40.568129+0000) 2024-03-26T02:38:56.043 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:55 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:55.987+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:26.469074+0000 front 2024-03-26T02:38:26.469103+0000 (oldest deadline 2024-03-26T02:38:52.269363+0000) 2024-03-26T02:38:56.313 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:56 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:56.042+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:38:56.313 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:56 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:56.042+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:25.673886+0000 front 2024-03-26T02:38:25.673968+0000 (oldest deadline 2024-03-26T02:38:50.373763+0000) 2024-03-26T02:38:57.054 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:56 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:56.767+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.726271+0000 front 2024-03-26T02:38:14.726289+0000 (oldest deadline 2024-03-26T02:38:43.526718+0000) 2024-03-26T02:38:57.054 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:56 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:56.767+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:23.527317+0000 front 2024-03-26T02:38:23.527304+0000 (oldest deadline 2024-03-26T02:38:48.827213+0000) 2024-03-26T02:38:57.054 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:56 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:56.994+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:15.868353+0000 front 2024-03-26T02:38:15.868227+0000 (oldest deadline 2024-03-26T02:38:40.568129+0000) 2024-03-26T02:38:57.054 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:56 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:56.994+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:26.469074+0000 front 2024-03-26T02:38:26.469103+0000 (oldest deadline 2024-03-26T02:38:52.269363+0000) 2024-03-26T02:38:57.054 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:56 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:56.994+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:33.370599+0000 front 2024-03-26T02:38:33.370575+0000 (oldest deadline 2024-03-26T02:38:56.870452+0000) 2024-03-26T02:38:57.313 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:57 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:57.052+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:38:57.313 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:57 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:57.052+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:25.673886+0000 front 2024-03-26T02:38:25.673968+0000 (oldest deadline 2024-03-26T02:38:50.373763+0000) 2024-03-26T02:38:58.063 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:57 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:57.801+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.726271+0000 front 2024-03-26T02:38:14.726289+0000 (oldest deadline 2024-03-26T02:38:43.526718+0000) 2024-03-26T02:38:58.063 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:57 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:57.801+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:23.527317+0000 front 2024-03-26T02:38:23.527304+0000 (oldest deadline 2024-03-26T02:38:48.827213+0000) 2024-03-26T02:38:58.064 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:58 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:58.028+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:38:58.064 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:58 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:58.028+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:25.673886+0000 front 2024-03-26T02:38:25.673968+0000 (oldest deadline 2024-03-26T02:38:50.373763+0000) 2024-03-26T02:38:58.064 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:57 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:57.956+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:15.868353+0000 front 2024-03-26T02:38:15.868227+0000 (oldest deadline 2024-03-26T02:38:40.568129+0000) 2024-03-26T02:38:58.064 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:57 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:57.956+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:26.469074+0000 front 2024-03-26T02:38:26.469103+0000 (oldest deadline 2024-03-26T02:38:52.269363+0000) 2024-03-26T02:38:58.064 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:57 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:57.956+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:33.370599+0000 front 2024-03-26T02:38:33.370575+0000 (oldest deadline 2024-03-26T02:38:56.870452+0000) 2024-03-26T02:38:59.063 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:58 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:58.788+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.726271+0000 front 2024-03-26T02:38:14.726289+0000 (oldest deadline 2024-03-26T02:38:43.526718+0000) 2024-03-26T02:38:59.063 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:58 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:58.788+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:23.527317+0000 front 2024-03-26T02:38:23.527304+0000 (oldest deadline 2024-03-26T02:38:48.827213+0000) 2024-03-26T02:38:59.064 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:58 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:58.955+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:15.868353+0000 front 2024-03-26T02:38:15.868227+0000 (oldest deadline 2024-03-26T02:38:40.568129+0000) 2024-03-26T02:38:59.064 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:58 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:58.955+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:26.469074+0000 front 2024-03-26T02:38:26.469103+0000 (oldest deadline 2024-03-26T02:38:52.269363+0000) 2024-03-26T02:38:59.064 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:38:58 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5[45248]: 2024-03-26T02:38:58.955+0000 7f1d71d50700 -1 osd.5 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:33.370599+0000 front 2024-03-26T02:38:33.370575+0000 (oldest deadline 2024-03-26T02:38:56.870452+0000) 2024-03-26T02:38:59.064 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:58 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:58.998+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:38:59.064 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:58 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:58.998+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:25.673886+0000 front 2024-03-26T02:38:25.673968+0000 (oldest deadline 2024-03-26T02:38:50.373763+0000) 2024-03-26T02:38:59.064 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:58 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:58.998+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:33.274770+0000 front 2024-03-26T02:38:33.274755+0000 (oldest deadline 2024-03-26T02:38:58.574582+0000) 2024-03-26T02:38:59.064 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:58 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:58.998+0000 7fceb9e15700 -1 osd.7 844 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:2225 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e844) 2024-03-26T02:39:00.088 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:59 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:59.814+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.726271+0000 front 2024-03-26T02:38:14.726289+0000 (oldest deadline 2024-03-26T02:38:43.526718+0000) 2024-03-26T02:39:00.088 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:59 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:59.814+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:23.527317+0000 front 2024-03-26T02:38:23.527304+0000 (oldest deadline 2024-03-26T02:38:48.827213+0000) 2024-03-26T02:39:00.088 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:38:59 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:38:59.814+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:32.928323+0000 front 2024-03-26T02:38:32.928338+0000 (oldest deadline 2024-03-26T02:38:58.828139+0000) 2024-03-26T02:39:00.088 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:59 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:59.958+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:39:00.088 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:59 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:59.958+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:25.673886+0000 front 2024-03-26T02:38:25.673968+0000 (oldest deadline 2024-03-26T02:38:50.373763+0000) 2024-03-26T02:39:00.088 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:59 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:59.958+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:33.274770+0000 front 2024-03-26T02:38:33.274755+0000 (oldest deadline 2024-03-26T02:38:58.574582+0000) 2024-03-26T02:39:00.088 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:38:59 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:38:59.958+0000 7fceb9e15700 -1 osd.7 844 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:2225 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e844) 2024-03-26T02:39:00.351 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:39:00 smithi173 podman[79792]: 2024-03-26 02:39:00.087376258 +0000 UTC m=+5.595618308 container died c245dc843a8e777e26da8402a8b96a78f6b8a3269e589ca66651854fa97c8fcd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-26T02:39:00.718 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:39:00 smithi173 podman[79792]: 2024-03-26 02:39:00.441891304 +0000 UTC m=+5.950133348 container cleanup c245dc843a8e777e26da8402a8b96a78f6b8a3269e589ca66651854fa97c8fcd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-26T02:39:00.718 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:39:00 smithi173 bash[79792]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5 2024-03-26T02:39:01.052 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:39:00 smithi173 podman[79814]: 2024-03-26 02:39:00.718086651 +0000 UTC m=+0.627415368 container remove c245dc843a8e777e26da8402a8b96a78f6b8a3269e589ca66651854fa97c8fcd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-5e70124, ceph=True, org.label-schema.license=GPLv2) 2024-03-26T02:39:01.053 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:00 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:00.817+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.726271+0000 front 2024-03-26T02:38:14.726289+0000 (oldest deadline 2024-03-26T02:38:43.526718+0000) 2024-03-26T02:39:01.053 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:00 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:00.817+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:23.527317+0000 front 2024-03-26T02:38:23.527304+0000 (oldest deadline 2024-03-26T02:38:48.827213+0000) 2024-03-26T02:39:01.053 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:00 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:00.817+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:32.928323+0000 front 2024-03-26T02:38:32.928338+0000 (oldest deadline 2024-03-26T02:38:58.828139+0000) 2024-03-26T02:39:01.053 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:00 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:00.940+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:39:01.053 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:00 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:00.940+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:25.673886+0000 front 2024-03-26T02:38:25.673968+0000 (oldest deadline 2024-03-26T02:38:50.373763+0000) 2024-03-26T02:39:01.053 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:00 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:00.940+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:33.274770+0000 front 2024-03-26T02:38:33.274755+0000 (oldest deadline 2024-03-26T02:38:58.574582+0000) 2024-03-26T02:39:01.053 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:00 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:00.940+0000 7fceb9e15700 -1 osd.7 844 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:2225 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e844) 2024-03-26T02:39:01.313 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:39:01 smithi173 podman[79909]: 2024-03-26 02:39:00.955601131 +0000 UTC m=+0.018106481 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:39:01.313 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:39:01 smithi173 podman[79909]: 2024-03-26 02:39:01.081077542 +0000 UTC m=+0.143582879 container create 37905c2529c161ea77de6520c291847128f32f51a2546b7f23e2df0e70a55e12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-26T02:39:01.313 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:39:01 smithi173 podman[79909]: 2024-03-26 02:39:01.27970387 +0000 UTC m=+0.342209207 container init 37905c2529c161ea77de6520c291847128f32f51a2546b7f23e2df0e70a55e12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-26T02:39:01.314 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:39:01 smithi173 podman[79909]: 2024-03-26 02:39:01.285185122 +0000 UTC m=+0.347690460 container start 37905c2529c161ea77de6520c291847128f32f51a2546b7f23e2df0e70a55e12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-26T02:39:01.813 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:39:01 smithi173 podman[79909]: 2024-03-26 02:39:01.362822185 +0000 UTC m=+0.425327544 container attach 37905c2529c161ea77de6520c291847128f32f51a2546b7f23e2df0e70a55e12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-26T02:39:01.813 INFO:journalctl@ceph.osd.5.smithi173.stdout:Mar 26 02:39:01 smithi173 podman[79909]: 2024-03-26 02:39:01.577274424 +0000 UTC m=+0.639779772 container died 37905c2529c161ea77de6520c291847128f32f51a2546b7f23e2df0e70a55e12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-26T02:39:02.191 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:01.842+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.726271+0000 front 2024-03-26T02:38:14.726289+0000 (oldest deadline 2024-03-26T02:38:43.526718+0000) 2024-03-26T02:39:02.191 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:01.842+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:23.527317+0000 front 2024-03-26T02:38:23.527304+0000 (oldest deadline 2024-03-26T02:38:48.827213+0000) 2024-03-26T02:39:02.191 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:01.842+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:32.928323+0000 front 2024-03-26T02:38:32.928338+0000 (oldest deadline 2024-03-26T02:38:58.828139+0000) 2024-03-26T02:39:02.192 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:01.989+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:39:02.192 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:01.989+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:25.673886+0000 front 2024-03-26T02:38:25.673968+0000 (oldest deadline 2024-03-26T02:38:50.373763+0000) 2024-03-26T02:39:02.192 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:01.989+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:33.274770+0000 front 2024-03-26T02:38:33.274755+0000 (oldest deadline 2024-03-26T02:38:58.574582+0000) 2024-03-26T02:39:02.192 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:01 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:01.989+0000 7fceb9e15700 -1 osd.7 844 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:2225 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e844) 2024-03-26T02:39:02.215 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.5.service' 2024-03-26T02:39:02.256 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:39:02.256 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-26T02:39:02.256 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-26T02:39:02.256 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.6 2024-03-26T02:39:02.563 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:02 smithi173 systemd[1]: Stopping Ceph osd.6 for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:39:02.821 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:02 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:02.563+0000 7f2df806b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:39:02.821 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:02 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:02.563+0000 7f2df806b700 -1 osd.6 844 *** Got signal Terminated *** 2024-03-26T02:39:02.821 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:02 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:02.563+0000 7f2df806b700 -1 osd.6 844 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:39:02.821 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:02 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:02.820+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.726271+0000 front 2024-03-26T02:38:14.726289+0000 (oldest deadline 2024-03-26T02:38:43.526718+0000) 2024-03-26T02:39:02.822 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:02 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:02.820+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:23.527317+0000 front 2024-03-26T02:38:23.527304+0000 (oldest deadline 2024-03-26T02:38:48.827213+0000) 2024-03-26T02:39:02.822 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:02 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:02.820+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:32.928323+0000 front 2024-03-26T02:38:32.928338+0000 (oldest deadline 2024-03-26T02:38:58.828139+0000) 2024-03-26T02:39:03.313 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:02 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:02.971+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:39:03.314 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:02 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:02.971+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:25.673886+0000 front 2024-03-26T02:38:25.673968+0000 (oldest deadline 2024-03-26T02:38:50.373763+0000) 2024-03-26T02:39:03.314 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:02 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:02.971+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:33.274770+0000 front 2024-03-26T02:38:33.274755+0000 (oldest deadline 2024-03-26T02:38:58.574582+0000) 2024-03-26T02:39:03.314 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:02 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:02.972+0000 7fceb9e15700 -1 osd.7 844 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:2225 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e844) 2024-03-26T02:39:04.313 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:03.848+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.726271+0000 front 2024-03-26T02:38:14.726289+0000 (oldest deadline 2024-03-26T02:38:43.526718+0000) 2024-03-26T02:39:04.313 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:03.848+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:23.527317+0000 front 2024-03-26T02:38:23.527304+0000 (oldest deadline 2024-03-26T02:38:48.827213+0000) 2024-03-26T02:39:04.313 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:03.848+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:32.928323+0000 front 2024-03-26T02:38:32.928338+0000 (oldest deadline 2024-03-26T02:38:58.828139+0000) 2024-03-26T02:39:04.314 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:03.951+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:39:04.314 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:03.951+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:25.673886+0000 front 2024-03-26T02:38:25.673968+0000 (oldest deadline 2024-03-26T02:38:50.373763+0000) 2024-03-26T02:39:04.314 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:03.951+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:33.274770+0000 front 2024-03-26T02:38:33.274755+0000 (oldest deadline 2024-03-26T02:38:58.574582+0000) 2024-03-26T02:39:04.314 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:03 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:03.951+0000 7fceb9e15700 -1 osd.7 844 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:2225 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e844) 2024-03-26T02:39:05.313 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:04 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:04.985+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:39:05.313 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:04 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:04.985+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:25.673886+0000 front 2024-03-26T02:38:25.673968+0000 (oldest deadline 2024-03-26T02:38:50.373763+0000) 2024-03-26T02:39:05.313 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:04 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:04.985+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:33.274770+0000 front 2024-03-26T02:38:33.274755+0000 (oldest deadline 2024-03-26T02:38:58.574582+0000) 2024-03-26T02:39:05.313 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:04 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:04.985+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6815 osd.3 since back 2024-03-26T02:38:42.075636+0000 front 2024-03-26T02:38:42.075579+0000 (oldest deadline 2024-03-26T02:39:04.875890+0000) 2024-03-26T02:39:05.313 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:04 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:04.985+0000 7fceb9e15700 -1 osd.7 844 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:2225 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e844) 2024-03-26T02:39:05.314 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:04 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:04.858+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.726271+0000 front 2024-03-26T02:38:14.726289+0000 (oldest deadline 2024-03-26T02:38:43.526718+0000) 2024-03-26T02:39:05.314 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:04 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:04.858+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:23.527317+0000 front 2024-03-26T02:38:23.527304+0000 (oldest deadline 2024-03-26T02:38:48.827213+0000) 2024-03-26T02:39:05.314 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:04 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:04.858+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:32.928323+0000 front 2024-03-26T02:38:32.928338+0000 (oldest deadline 2024-03-26T02:38:58.828139+0000) 2024-03-26T02:39:05.314 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:04 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:04.858+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6815 osd.3 since back 2024-03-26T02:38:40.429669+0000 front 2024-03-26T02:38:40.429640+0000 (oldest deadline 2024-03-26T02:39:04.529599+0000) 2024-03-26T02:39:06.313 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:05 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:05.900+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.726271+0000 front 2024-03-26T02:38:14.726289+0000 (oldest deadline 2024-03-26T02:38:43.526718+0000) 2024-03-26T02:39:06.313 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:05 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:05.900+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:23.527317+0000 front 2024-03-26T02:38:23.527304+0000 (oldest deadline 2024-03-26T02:38:48.827213+0000) 2024-03-26T02:39:06.314 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:05 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:05.900+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:32.928323+0000 front 2024-03-26T02:38:32.928338+0000 (oldest deadline 2024-03-26T02:38:58.828139+0000) 2024-03-26T02:39:06.314 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:05 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:05.900+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6815 osd.3 since back 2024-03-26T02:38:40.429669+0000 front 2024-03-26T02:38:40.429640+0000 (oldest deadline 2024-03-26T02:39:04.529599+0000) 2024-03-26T02:39:06.314 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:06 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:06.001+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:39:06.314 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:06 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:06.001+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:25.673886+0000 front 2024-03-26T02:38:25.673968+0000 (oldest deadline 2024-03-26T02:38:50.373763+0000) 2024-03-26T02:39:06.314 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:06 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:06.001+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:33.274770+0000 front 2024-03-26T02:38:33.274755+0000 (oldest deadline 2024-03-26T02:38:58.574582+0000) 2024-03-26T02:39:06.314 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:06 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:06.001+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6815 osd.3 since back 2024-03-26T02:38:42.075636+0000 front 2024-03-26T02:38:42.075579+0000 (oldest deadline 2024-03-26T02:39:04.875890+0000) 2024-03-26T02:39:06.314 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:06 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:06.002+0000 7fceb9e15700 -1 osd.7 844 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:2225 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e844) 2024-03-26T02:39:07.313 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:06 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:06.919+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.726271+0000 front 2024-03-26T02:38:14.726289+0000 (oldest deadline 2024-03-26T02:38:43.526718+0000) 2024-03-26T02:39:07.313 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:06 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:06.919+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:23.527317+0000 front 2024-03-26T02:38:23.527304+0000 (oldest deadline 2024-03-26T02:38:48.827213+0000) 2024-03-26T02:39:07.313 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:06 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:06.919+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:32.928323+0000 front 2024-03-26T02:38:32.928338+0000 (oldest deadline 2024-03-26T02:38:58.828139+0000) 2024-03-26T02:39:07.313 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:06 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6[51638]: 2024-03-26T02:39:06.919+0000 7f2df465f700 -1 osd.6 844 heartbeat_check: no reply from 172.21.15.72:6815 osd.3 since back 2024-03-26T02:38:40.429669+0000 front 2024-03-26T02:38:40.429640+0000 (oldest deadline 2024-03-26T02:39:04.529599+0000) 2024-03-26T02:39:07.314 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:06 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:06.982+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:39:07.314 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:06 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:06.982+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:25.673886+0000 front 2024-03-26T02:38:25.673968+0000 (oldest deadline 2024-03-26T02:38:50.373763+0000) 2024-03-26T02:39:07.314 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:06 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:06.982+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:33.274770+0000 front 2024-03-26T02:38:33.274755+0000 (oldest deadline 2024-03-26T02:38:58.574582+0000) 2024-03-26T02:39:07.314 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:06 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:06.982+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6815 osd.3 since back 2024-03-26T02:38:42.075636+0000 front 2024-03-26T02:38:42.075579+0000 (oldest deadline 2024-03-26T02:39:04.875890+0000) 2024-03-26T02:39:07.314 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:06 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:06.982+0000 7fceb9e15700 -1 osd.7 844 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:2225 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e844) 2024-03-26T02:39:08.278 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:08 smithi173 podman[80042]: 2024-03-26 02:39:08.002962073 +0000 UTC m=+5.528880425 container died 37382c9dfeb72c925fe847efbdb2b99c6bab21ad0abd66da3af1455f0451ff08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:39:08.278 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:08 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:08.000+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:39:08.279 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:08 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:08.000+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:25.673886+0000 front 2024-03-26T02:38:25.673968+0000 (oldest deadline 2024-03-26T02:38:50.373763+0000) 2024-03-26T02:39:08.279 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:08 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:08.000+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:33.274770+0000 front 2024-03-26T02:38:33.274755+0000 (oldest deadline 2024-03-26T02:38:58.574582+0000) 2024-03-26T02:39:08.279 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:08 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:08.000+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6815 osd.3 since back 2024-03-26T02:38:42.075636+0000 front 2024-03-26T02:38:42.075579+0000 (oldest deadline 2024-03-26T02:39:04.875890+0000) 2024-03-26T02:39:08.279 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:08 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:08.001+0000 7fceb9e15700 -1 osd.7 844 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:2225 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e844) 2024-03-26T02:39:08.544 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:08 smithi173 podman[80042]: 2024-03-26 02:39:08.343586982 +0000 UTC m=+5.869505334 container cleanup 37382c9dfeb72c925fe847efbdb2b99c6bab21ad0abd66da3af1455f0451ff08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-26T02:39:08.545 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:08 smithi173 bash[80042]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6 2024-03-26T02:39:08.813 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:08 smithi173 podman[80066]: 2024-03-26 02:39:08.54462324 +0000 UTC m=+0.541695750 container remove 37382c9dfeb72c925fe847efbdb2b99c6bab21ad0abd66da3af1455f0451ff08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124) 2024-03-26T02:39:09.281 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:09 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:09.017+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:39:09.281 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:09 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:09.017+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:25.673886+0000 front 2024-03-26T02:38:25.673968+0000 (oldest deadline 2024-03-26T02:38:50.373763+0000) 2024-03-26T02:39:09.281 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:09 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:09.017+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:33.274770+0000 front 2024-03-26T02:38:33.274755+0000 (oldest deadline 2024-03-26T02:38:58.574582+0000) 2024-03-26T02:39:09.282 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:09 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:09.017+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6815 osd.3 since back 2024-03-26T02:38:42.075636+0000 front 2024-03-26T02:38:42.075579+0000 (oldest deadline 2024-03-26T02:39:04.875890+0000) 2024-03-26T02:39:09.282 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:09 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:09.017+0000 7fceb9e15700 -1 osd.7 844 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:2225 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e844) 2024-03-26T02:39:09.282 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:08 smithi173 podman[80164]: 2024-03-26 02:39:08.868780131 +0000 UTC m=+0.017973803 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:39:09.282 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:08 smithi173 podman[80164]: 2024-03-26 02:39:08.970673582 +0000 UTC m=+0.119867292 container create 628397b61f8c08c535e0e37ce4fc1af666a7252cde199b27537904cbe916e9d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6-deactivate, RELEASE=squid-5e70124, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T02:39:09.282 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:09 smithi173 podman[80164]: 2024-03-26 02:39:09.173502516 +0000 UTC m=+0.322696196 container init 628397b61f8c08c535e0e37ce4fc1af666a7252cde199b27537904cbe916e9d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-26T02:39:09.282 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:09 smithi173 podman[80164]: 2024-03-26 02:39:09.179537069 +0000 UTC m=+0.328730738 container start 628397b61f8c08c535e0e37ce4fc1af666a7252cde199b27537904cbe916e9d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-26T02:39:09.282 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:09 smithi173 podman[80164]: 2024-03-26 02:39:09.281240858 +0000 UTC m=+0.430434538 container attach 628397b61f8c08c535e0e37ce4fc1af666a7252cde199b27537904cbe916e9d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124) 2024-03-26T02:39:09.563 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:09 smithi173 podman[80164]: 2024-03-26 02:39:09.470826881 +0000 UTC m=+0.620020558 container died 628397b61f8c08c535e0e37ce4fc1af666a7252cde199b27537904cbe916e9d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124) 2024-03-26T02:39:10.001 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:09 smithi173 podman[80209]: 2024-03-26 02:39:09.984293815 +0000 UTC m=+0.508581849 container remove 628397b61f8c08c535e0e37ce4fc1af666a7252cde199b27537904cbe916e9d0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-6-deactivate, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-26T02:39:10.001 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:09 smithi173 systemd[1]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.6.service: Deactivated successfully. 2024-03-26T02:39:10.001 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:09 smithi173 systemd[1]: Stopped Ceph osd.6 for c86fbdd4-eb13-11ee-95d0-87774f69a715. 2024-03-26T02:39:10.001 INFO:journalctl@ceph.osd.6.smithi173.stdout:Mar 26 02:39:10 smithi173 systemd[1]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.6.service: Consumed 53.611s CPU time. 2024-03-26T02:39:10.002 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.6.service' 2024-03-26T02:39:10.047 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:39:10.047 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-26T02:39:10.047 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-26T02:39:10.047 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.7 2024-03-26T02:39:10.313 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:10 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:10.027+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:39:10.314 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:10 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:10.027+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:25.673886+0000 front 2024-03-26T02:38:25.673968+0000 (oldest deadline 2024-03-26T02:38:50.373763+0000) 2024-03-26T02:39:10.314 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:10 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:10.027+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:33.274770+0000 front 2024-03-26T02:38:33.274755+0000 (oldest deadline 2024-03-26T02:38:58.574582+0000) 2024-03-26T02:39:10.314 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:10 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:10.027+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6815 osd.3 since back 2024-03-26T02:38:42.075636+0000 front 2024-03-26T02:38:42.075579+0000 (oldest deadline 2024-03-26T02:39:04.875890+0000) 2024-03-26T02:39:10.314 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:10 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:10.027+0000 7fceb9e15700 -1 osd.7 844 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:2225 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e844) 2024-03-26T02:39:10.314 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:10 smithi173 systemd[1]: Stopping Ceph osd.7 for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:39:10.813 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:10 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:10.373+0000 7fcebd821700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:39:10.813 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:10 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:10.373+0000 7fcebd821700 -1 osd.7 844 *** Got signal Terminated *** 2024-03-26T02:39:10.813 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:10 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:10.373+0000 7fcebd821700 -1 osd.7 844 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T02:39:11.563 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:11 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:11.074+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:39:11.563 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:11 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:11.074+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:25.673886+0000 front 2024-03-26T02:38:25.673968+0000 (oldest deadline 2024-03-26T02:38:50.373763+0000) 2024-03-26T02:39:11.563 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:11 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:11.074+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:33.274770+0000 front 2024-03-26T02:38:33.274755+0000 (oldest deadline 2024-03-26T02:38:58.574582+0000) 2024-03-26T02:39:11.564 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:11 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:11.074+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6815 osd.3 since back 2024-03-26T02:38:42.075636+0000 front 2024-03-26T02:38:42.075579+0000 (oldest deadline 2024-03-26T02:39:04.875890+0000) 2024-03-26T02:39:11.564 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:11 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:11.074+0000 7fceb9e15700 -1 osd.7 844 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:2225 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e844) 2024-03-26T02:39:12.465 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:12 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:12.107+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:39:12.465 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:12 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:12.107+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:25.673886+0000 front 2024-03-26T02:38:25.673968+0000 (oldest deadline 2024-03-26T02:38:50.373763+0000) 2024-03-26T02:39:12.466 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:12 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:12.107+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:33.274770+0000 front 2024-03-26T02:38:33.274755+0000 (oldest deadline 2024-03-26T02:38:58.574582+0000) 2024-03-26T02:39:12.466 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:12 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:12.107+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6815 osd.3 since back 2024-03-26T02:38:42.075636+0000 front 2024-03-26T02:38:42.075579+0000 (oldest deadline 2024-03-26T02:39:04.875890+0000) 2024-03-26T02:39:12.466 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:12 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:12.107+0000 7fceb9e15700 -1 osd.7 844 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:2225 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e844) 2024-03-26T02:39:13.477 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:13 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:13.109+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:39:13.477 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:13 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:13.109+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:25.673886+0000 front 2024-03-26T02:38:25.673968+0000 (oldest deadline 2024-03-26T02:38:50.373763+0000) 2024-03-26T02:39:13.477 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:13 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:13.109+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:33.274770+0000 front 2024-03-26T02:38:33.274755+0000 (oldest deadline 2024-03-26T02:38:58.574582+0000) 2024-03-26T02:39:13.477 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:13 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:13.109+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6815 osd.3 since back 2024-03-26T02:38:42.075636+0000 front 2024-03-26T02:38:42.075579+0000 (oldest deadline 2024-03-26T02:39:04.875890+0000) 2024-03-26T02:39:13.477 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:13 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:13.109+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.173:6802 osd.4 since back 2024-03-26T02:38:50.776925+0000 front 2024-03-26T02:38:50.776869+0000 (oldest deadline 2024-03-26T02:39:12.476837+0000) 2024-03-26T02:39:13.477 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:13 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:13.109+0000 7fceb9e15700 -1 osd.7 844 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:2225 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e844) 2024-03-26T02:39:14.563 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:14 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:14.131+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:39:14.563 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:14 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:14.131+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:25.673886+0000 front 2024-03-26T02:38:25.673968+0000 (oldest deadline 2024-03-26T02:38:50.373763+0000) 2024-03-26T02:39:14.564 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:14 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:14.131+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:33.274770+0000 front 2024-03-26T02:38:33.274755+0000 (oldest deadline 2024-03-26T02:38:58.574582+0000) 2024-03-26T02:39:14.564 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:14 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:14.131+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6815 osd.3 since back 2024-03-26T02:38:42.075636+0000 front 2024-03-26T02:38:42.075579+0000 (oldest deadline 2024-03-26T02:39:04.875890+0000) 2024-03-26T02:39:14.564 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:14 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:14.131+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.173:6802 osd.4 since back 2024-03-26T02:38:50.776925+0000 front 2024-03-26T02:38:50.776869+0000 (oldest deadline 2024-03-26T02:39:12.476837+0000) 2024-03-26T02:39:14.564 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:14 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:14.132+0000 7fceb9e15700 -1 osd.7 844 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:2225 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e844) 2024-03-26T02:39:15.563 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:15 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:15.096+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6803 osd.0 since back 2024-03-26T02:38:14.072058+0000 front 2024-03-26T02:38:14.072105+0000 (oldest deadline 2024-03-26T02:38:42.272304+0000) 2024-03-26T02:39:15.563 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:15 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:15.096+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6807 osd.1 since back 2024-03-26T02:38:25.673886+0000 front 2024-03-26T02:38:25.673968+0000 (oldest deadline 2024-03-26T02:38:50.373763+0000) 2024-03-26T02:39:15.563 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:15 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:15.096+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6811 osd.2 since back 2024-03-26T02:38:33.274770+0000 front 2024-03-26T02:38:33.274755+0000 (oldest deadline 2024-03-26T02:38:58.574582+0000) 2024-03-26T02:39:15.564 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:15 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:15.096+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.72:6815 osd.3 since back 2024-03-26T02:38:42.075636+0000 front 2024-03-26T02:38:42.075579+0000 (oldest deadline 2024-03-26T02:39:04.875890+0000) 2024-03-26T02:39:15.564 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:15 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:15.096+0000 7fceb9e15700 -1 osd.7 844 heartbeat_check: no reply from 172.21.15.173:6802 osd.4 since back 2024-03-26T02:38:50.776925+0000 front 2024-03-26T02:38:50.776869+0000 (oldest deadline 2024-03-26T02:39:12.476837+0000) 2024-03-26T02:39:15.564 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:15 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7[57724]: 2024-03-26T02:39:15.096+0000 7fceb9e15700 -1 osd.7 844 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24409.0:2225 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e844) 2024-03-26T02:39:16.038 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:15 smithi173 podman[80296]: 2024-03-26 02:39:15.771389957 +0000 UTC m=+5.500936451 container died caa26bfa8992a39b30e12037f7d3b4d3cd9ee9d88ae9a13bc897a30f75d4d69e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T02:39:16.313 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:16 smithi173 podman[80296]: 2024-03-26 02:39:16.128423371 +0000 UTC m=+5.857969865 container cleanup caa26bfa8992a39b30e12037f7d3b4d3cd9ee9d88ae9a13bc897a30f75d4d69e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=) 2024-03-26T02:39:16.313 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:16 smithi173 bash[80296]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7 2024-03-26T02:39:16.800 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:16 smithi173 podman[80330]: 2024-03-26 02:39:16.446390398 +0000 UTC m=+0.667364557 container remove caa26bfa8992a39b30e12037f7d3b4d3cd9ee9d88ae9a13bc897a30f75d4d69e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-26T02:39:17.055 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:16 smithi173 podman[80428]: 2024-03-26 02:39:16.702623127 +0000 UTC m=+0.017498106 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T02:39:17.055 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:16 smithi173 podman[80428]: 2024-03-26 02:39:16.813852532 +0000 UTC m=+0.128727498 container create b30e5340039184e212ebd3a9cd9a856445bbb881c5252dfff835744e7e4647bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True) 2024-03-26T02:39:17.056 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:17 smithi173 podman[80428]: 2024-03-26 02:39:17.049619751 +0000 UTC m=+0.364494718 container init b30e5340039184e212ebd3a9cd9a856445bbb881c5252dfff835744e7e4647bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-26T02:39:17.056 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:17 smithi173 podman[80428]: 2024-03-26 02:39:17.055359557 +0000 UTC m=+0.370234523 container start b30e5340039184e212ebd3a9cd9a856445bbb881c5252dfff835744e7e4647bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-26T02:39:17.313 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:17 smithi173 podman[80428]: 2024-03-26 02:39:17.157966729 +0000 UTC m=+0.472841697 container attach b30e5340039184e212ebd3a9cd9a856445bbb881c5252dfff835744e7e4647bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7-deactivate, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-26T02:39:17.636 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:17 smithi173 podman[80428]: 2024-03-26 02:39:17.341854181 +0000 UTC m=+0.656729148 container died b30e5340039184e212ebd3a9cd9a856445bbb881c5252dfff835744e7e4647bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-26T02:39:17.888 INFO:journalctl@ceph.osd.7.smithi173.stdout:Mar 26 02:39:17 smithi173 podman[80474]: 2024-03-26 02:39:17.869209272 +0000 UTC m=+0.521640296 container remove b30e5340039184e212ebd3a9cd9a856445bbb881c5252dfff835744e7e4647bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-26T02:39:17.891 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@osd.7.service' 2024-03-26T02:39:17.933 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:39:17.933 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-26T02:39:17.933 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-26T02:39:17.933 DEBUG:teuthology.orchestra.run.smithi072:> sudo systemctl stop ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@rgw.foo.a 2024-03-26T02:39:18.219 INFO:journalctl@ceph.rgw.foo.a.smithi072.stdout:Mar 26 02:39:17 smithi072 systemd[1]: Stopping Ceph rgw.foo.a for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:39:18.483 INFO:journalctl@ceph.rgw.foo.a.smithi072.stdout:Mar 26 02:39:18 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-rgw-foo-a[70518]: 2024-03-26T02:39:18.231+0000 7f590324a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T02:39:18.483 INFO:journalctl@ceph.rgw.foo.a.smithi072.stdout:Mar 26 02:39:18 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-rgw-foo-a[70518]: 2024-03-26T02:39:18.231+0000 7f5911ac7ac0 -1 shutting down 2024-03-26T02:39:26.063 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:39:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:39:25.619Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.72:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.72:8765: connect: connection refused" 2024-03-26T02:39:26.064 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:39:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:39:25.619Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.72:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.72:8765: connect: connection refused" 2024-03-26T02:39:26.064 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:39:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:39:25.619Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.72:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.72:8765: connect: connection refused" 2024-03-26T02:39:26.064 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:39:25 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:39:25.622Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.72:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.72:8765: connect: connection refused" 2024-03-26T02:39:28.514 INFO:journalctl@ceph.rgw.foo.a.smithi072.stdout:Mar 26 02:39:28 smithi072 bash[161845]: time="2024-03-26T02:39:28Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-26T02:39:28.515 INFO:journalctl@ceph.rgw.foo.a.smithi072.stdout:Mar 26 02:39:28 smithi072 podman[161845]: 2024-03-26 02:39:28.25223077 +0000 UTC m=+10.128736620 container died 6ef2c13736317266c695bfb57d41686b431bb699f2bb158de55fbf2986bb7a29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-rgw-foo-a, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-26T02:39:28.884 INFO:journalctl@ceph.rgw.foo.a.smithi072.stdout:Mar 26 02:39:28 smithi072 podman[161845]: 2024-03-26 02:39:28.613807578 +0000 UTC m=+10.490313437 container cleanup 6ef2c13736317266c695bfb57d41686b431bb699f2bb158de55fbf2986bb7a29 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-rgw-foo-a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-26T02:39:28.884 INFO:journalctl@ceph.rgw.foo.a.smithi072.stdout:Mar 26 02:39:28 smithi072 bash[161845]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-rgw-foo-a 2024-03-26T02:39:29.035 DEBUG:teuthology.orchestra.run.smithi072:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@rgw.foo.a.service' 2024-03-26T02:39:29.074 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:39:29.074 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-26T02:39:29.074 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-26T02:39:29.074 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@prometheus.a 2024-03-26T02:39:29.390 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:39:29 smithi173 systemd[1]: Stopping Ceph prometheus.a for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:39:29.691 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:39:29 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:39:29.390Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T02:39:29.691 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:39:29 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:39:29.390Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-26T02:39:29.691 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:39:29 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:39:29.390Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-26T02:39:29.691 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:39:29 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:39:29.390Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T02:39:29.691 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:39:29 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:39:29.390Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-26T02:39:29.691 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:39:29 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:39:29.390Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-26T02:39:29.691 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:39:29 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:39:29.390Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T02:39:29.691 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:39:29 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:39:29.390Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-26T02:39:29.691 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:39:29 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:39:29.390Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-26T02:39:29.691 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:39:29 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:39:29.391Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T02:39:29.692 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:39:29 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:39:29.391Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-26T02:39:29.692 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:39:29 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a[68270]: ts=2024-03-26T02:39:29.391Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-26T02:39:29.692 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:39:29 smithi173 podman[80559]: 2024-03-26 02:39:29.400908496 +0000 UTC m=+0.130730482 container died 25ef588f85b82c172031eb42c1e78a51458adec36ca51a90656eb4eec527ada1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:39:30.030 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:39:29 smithi173 podman[80559]: 2024-03-26 02:39:29.787676536 +0000 UTC m=+0.517498524 container cleanup 25ef588f85b82c172031eb42c1e78a51458adec36ca51a90656eb4eec527ada1 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T02:39:30.030 INFO:journalctl@ceph.prometheus.a.smithi173.stdout:Mar 26 02:39:29 smithi173 bash[80559]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T02:39:30.150 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@prometheus.a.service' 2024-03-26T02:39:30.187 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T02:39:30.188 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-26T02:39:30.188 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm rm-cluster --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T02:39:30.371 INFO:teuthology.orchestra.run.smithi072.stdout:Deleting cluster with fsid: c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T02:39:32.151 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:39:31 smithi072 systemd[1]: Stopping Ceph alertmanager.a for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:39:32.468 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:39:32 smithi072 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a[81552]: ts=2024-03-26T02:39:32.151Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-26T02:39:32.468 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:39:32 smithi072 podman[162364]: 2024-03-26 02:39:32.182244086 +0000 UTC m=+0.132026660 container died 5ac26ae8cdb85eab56cfd04168026ac56098d12a240f90d46667f8df0a7f3c7c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:39:32.719 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:39:32 smithi072 podman[162364]: 2024-03-26 02:39:32.577309429 +0000 UTC m=+0.527092001 container cleanup 5ac26ae8cdb85eab56cfd04168026ac56098d12a240f90d46667f8df0a7f3c7c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:39:32.719 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:39:32 smithi072 bash[162364]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T02:39:33.219 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:39:32 smithi072 podman[162383]: 2024-03-26 02:39:32.81304674 +0000 UTC m=+0.624685354 container remove 5ac26ae8cdb85eab56cfd04168026ac56098d12a240f90d46667f8df0a7f3c7c (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T02:39:33.219 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:39:32 smithi072 podman[162383]: 2024-03-26 02:39:32.880359045 +0000 UTC m=+0.691997614 volume remove 306385f69bcab269c1dddabcf459af04dde6b15d3226e44db036681fe7163cf6 2024-03-26T02:39:33.219 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:39:33 smithi072 systemd[1]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T02:39:33.219 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:39:33 smithi072 systemd[1]: Stopped Ceph alertmanager.a for c86fbdd4-eb13-11ee-95d0-87774f69a715. 2024-03-26T02:39:33.219 INFO:journalctl@ceph.alertmanager.a.smithi072.stdout:Mar 26 02:39:33 smithi072 systemd[1]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@alertmanager.a.service: Consumed 2.530s CPU time. 2024-03-26T02:39:33.719 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:39:33 smithi072 systemd[1]: Stopping Ceph node-exporter.a for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:39:33.719 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:39:33 smithi072 podman[162499]: 2024-03-26 02:39:33.480819664 +0000 UTC m=+0.108253684 container died 2568a301a2cb4316d62b6ad51b7630db732204fc359224994f39c41961fdef52 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:39:34.081 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:39:33 smithi072 podman[162499]: 2024-03-26 02:39:33.829660208 +0000 UTC m=+0.457094226 container cleanup 2568a301a2cb4316d62b6ad51b7630db732204fc359224994f39c41961fdef52 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:39:34.081 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:39:33 smithi072 bash[162499]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a 2024-03-26T02:39:34.381 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:39:34 smithi072 podman[162518]: 2024-03-26 02:39:34.081121594 +0000 UTC m=+0.604978489 container remove 2568a301a2cb4316d62b6ad51b7630db732204fc359224994f39c41961fdef52 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T02:39:34.381 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:39:34 smithi072 systemd[1]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-26T02:39:34.381 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:39:34 smithi072 systemd[1]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-26T02:39:34.381 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:39:34 smithi072 systemd[1]: Stopped Ceph node-exporter.a for c86fbdd4-eb13-11ee-95d0-87774f69a715. 2024-03-26T02:39:34.381 INFO:journalctl@ceph.node-exporter.a.smithi072.stdout:Mar 26 02:39:34 smithi072 systemd[1]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@node-exporter.a.service: Consumed 5.205s CPU time. 2024-03-26T02:39:34.856 DEBUG:teuthology.orchestra.run.smithi173:> sudo cephadm rm-cluster --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T02:39:35.047 INFO:teuthology.orchestra.run.smithi173.stdout:Deleting cluster with fsid: c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T02:39:36.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:39:36 smithi173 systemd[1]: Stopping Ceph iscsi.iscsi.a for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:39:36.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:39:36 smithi173 ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a[62846]: debug Shutdown received 2024-03-26T02:39:46.814 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:39:46 smithi173 bash[80982]: time="2024-03-26T02:39:46Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-26T02:39:46.814 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:39:46 smithi173 podman[80982]: 2024-03-26 02:39:46.555164018 +0000 UTC m=+10.126508808 container died 3a58dadfedfafb6e9796d22a7868e28822528a3dae2dfb5b070b68c02a0e413f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-03-26T02:39:47.234 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:39:46 smithi173 podman[80982]: 2024-03-26 02:39:46.990362047 +0000 UTC m=+10.561706833 container cleanup 3a58dadfedfafb6e9796d22a7868e28822528a3dae2dfb5b070b68c02a0e413f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-26T02:39:47.234 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:39:46 smithi173 bash[80982]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a 2024-03-26T02:39:47.485 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:39:47 smithi173 podman[81008]: 2024-03-26 02:39:47.233471033 +0000 UTC m=+0.675068291 container remove 3a58dadfedfafb6e9796d22a7868e28822528a3dae2dfb5b070b68c02a0e413f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-iscsi-iscsi-a, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, GIT_BRANCH=HEAD) 2024-03-26T02:39:47.485 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:39:47 smithi173 systemd[1]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-26T02:39:47.485 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:39:47 smithi173 systemd[1]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-26T02:39:47.485 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:39:47 smithi173 systemd[1]: Stopped Ceph iscsi.iscsi.a for c86fbdd4-eb13-11ee-95d0-87774f69a715. 2024-03-26T02:39:47.485 INFO:journalctl@ceph.iscsi.iscsi.a.smithi173.stdout:Mar 26 02:39:47 smithi173 systemd[1]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Consumed 4.428s CPU time. 2024-03-26T02:39:49.063 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:39:48 smithi173 systemd[1]: Stopping Ceph grafana.a for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:39:49.424 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:39:49 smithi173 podman[81291]: 2024-03-26 02:39:49.16230218 +0000 UTC m=+0.113391565 container died 46d8593c9f17ea9e200b1c07b724f77b9a660540bc02c29658c4cf2ef80d3dbe (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-04-29T11:48:11.055897, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, architecture=x86_64, io.buildah.version=1.32.0, name=ubi8, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , version=8.5, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664) 2024-03-26T02:39:49.682 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:39:49 smithi173 podman[81291]: 2024-03-26 02:39:49.498127548 +0000 UTC m=+0.449216938 container cleanup 46d8593c9f17ea9e200b1c07b724f77b9a660540bc02c29658c4cf2ef80d3dbe (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a, io.openshift.expose-services=, release=239.1651231664, io.buildah.version=1.32.0, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, name=ubi8, architecture=x86_64, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, distribution-scope=public, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, com.redhat.component=ubi8-container) 2024-03-26T02:39:49.682 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:39:49 smithi173 bash[81291]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a 2024-03-26T02:39:50.063 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:39:49 smithi173 podman[81310]: 2024-03-26 02:39:49.682186178 +0000 UTC m=+0.519279607 container remove 46d8593c9f17ea9e200b1c07b724f77b9a660540bc02c29658c4cf2ef80d3dbe (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-grafana-a, name=ubi8, release=239.1651231664, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, description=Ceph Grafana Container, vcs-type=git, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, io.buildah.version=1.32.0, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, maintainer=Paul Cuzner , vendor=Red Hat, Inc.) 2024-03-26T02:39:50.063 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:39:49 smithi173 systemd[1]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-26T02:39:50.063 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:39:49 smithi173 systemd[1]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-26T02:39:50.063 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:39:49 smithi173 systemd[1]: Stopped Ceph grafana.a for c86fbdd4-eb13-11ee-95d0-87774f69a715. 2024-03-26T02:39:50.063 INFO:journalctl@ceph.grafana.a.smithi173.stdout:Mar 26 02:39:49 smithi173 systemd[1]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@grafana.a.service: Consumed 18.319s CPU time. 2024-03-26T02:39:50.563 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:39:50 smithi173 systemd[1]: Stopping Ceph node-exporter.b for c86fbdd4-eb13-11ee-95d0-87774f69a715... 2024-03-26T02:39:50.563 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:39:50 smithi173 podman[81426]: 2024-03-26 02:39:50.346799278 +0000 UTC m=+0.098471733 container died 107338ad9b8535493bb7b1dbc80d15b2fb55ebd87d8f6e068ce93b03cac47345 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:39:50.867 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:39:50 smithi173 podman[81426]: 2024-03-26 02:39:50.674999754 +0000 UTC m=+0.426672191 container cleanup 107338ad9b8535493bb7b1dbc80d15b2fb55ebd87d8f6e068ce93b03cac47345 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:39:50.867 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:39:50 smithi173 bash[81426]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b 2024-03-26T02:39:51.230 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:39:50 smithi173 podman[81445]: 2024-03-26 02:39:50.867660322 +0000 UTC m=+0.525778607 container remove 107338ad9b8535493bb7b1dbc80d15b2fb55ebd87d8f6e068ce93b03cac47345 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T02:39:51.230 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:39:50 smithi173 systemd[1]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-26T02:39:51.230 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:39:51 smithi173 systemd[1]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-26T02:39:51.230 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:39:51 smithi173 systemd[1]: Stopped Ceph node-exporter.b for c86fbdd4-eb13-11ee-95d0-87774f69a715. 2024-03-26T02:39:51.231 INFO:journalctl@ceph.node-exporter.b.smithi173.stdout:Mar 26 02:39:51 smithi173 systemd[1]: ceph-c86fbdd4-eb13-11ee-95d0-87774f69a715@node-exporter.b.service: Consumed 5.225s CPU time. 2024-03-26T02:39:51.716 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:39:51.751 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T02:39:51.781 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-26T02:39:51.783 DEBUG:teuthology.misc:Transferring archived files from smithi072:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621931/remote/smithi072/crash 2024-03-26T02:39:51.784 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/crash -- . 2024-03-26T02:39:51.835 INFO:teuthology.orchestra.run.smithi072.stderr:tar: /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T02:39:51.835 INFO:teuthology.orchestra.run.smithi072.stderr:tar: Error is not recoverable: exiting now 2024-03-26T02:39:51.837 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621931/remote/smithi173/crash 2024-03-26T02:39:51.838 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/crash -- . 2024-03-26T02:39:51.875 INFO:teuthology.orchestra.run.smithi173.stderr:tar: /var/lib/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T02:39:51.876 INFO:teuthology.orchestra.run.smithi173.stderr:tar: Error is not recoverable: exiting now 2024-03-26T02:39:51.877 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-26T02:39:51.877 DEBUG:teuthology.orchestra.run.smithi072:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-26T02:39:51.911 INFO:teuthology.orchestra.run.smithi072.stdout:2024-03-26T02:10:00.000151+0000 mon.a (mon.0) 2538 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 4 pool(s) do not have an application enabled 2024-03-26T02:39:51.911 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-26T02:39:51.911 DEBUG:teuthology.orchestra.run.smithi072:> sudo egrep '\[SEC\]' /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-26T02:39:51.984 DEBUG:teuthology.orchestra.run.smithi072:> sudo egrep '\[ERR\]' /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-26T02:39:52.055 DEBUG:teuthology.orchestra.run.smithi072:> sudo egrep '\[WRN\]' /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-26T02:39:52.126 INFO:teuthology.orchestra.run.smithi072.stdout:2024-03-26T02:10:00.000151+0000 mon.a (mon.0) 2538 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 4 pool(s) do not have an application enabled 2024-03-26T02:39:52.127 INFO:tasks.cephadm:Compressing logs... 2024-03-26T02:39:52.127 DEBUG:teuthology.orchestra.run.smithi072:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T02:39:52.169 DEBUG:teuthology.orchestra.run.smithi173:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T02:39:52.193 INFO:teuthology.orchestra.run.smithi072.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T02:39:52.193 INFO:teuthology.orchestra.run.smithi072.stderr:: No such file or directory 2024-03-26T02:39:52.193 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-mon.a.log 2024-03-26T02:39:52.193 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T02:39:52.195 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-26T02:39:52.195 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-mon.a.log: /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-osd.1.log 2024-03-26T02:39:52.195 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-osd.3.log 2024-03-26T02:39:52.195 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-mgr.y.log 2024-03-26T02:39:52.195 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-26T02:39:52.196 INFO:teuthology.orchestra.run.smithi173.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T02:39:52.196 INFO:teuthology.orchestra.run.smithi173.stderr:: No such file or directory 2024-03-26T02:39:52.197 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-osd.6.log 2024-03-26T02:39:52.197 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T02:39:52.197 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-osd.4.log 2024-03-26T02:39:52.197 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-osd.7.log 2024-03-26T02:39:52.202 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-osd.5.log 2024-03-26T02:39:52.202 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-osd.4.log: /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-mgr.x.log 2024-03-26T02:39:52.202 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/tcmu-runner.log 2024-03-26T02:39:52.202 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T02:39:52.202 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/tcmu-runner.log: 68.0% -- replaced with /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/tcmu-runner.log.gz 2024-03-26T02:39:52.202 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph.log 2024-03-26T02:39:52.202 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph.cephadm.log: 81.6% -- replaced with /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T02:39:52.202 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T02:39:52.202 INFO:teuthology.orchestra.run.smithi173.stderr: 85.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T02:39:52.202 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-mon.b.log 2024-03-26T02:39:52.204 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-mgr.y.log: 93.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T02:39:52.204 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T02:39:52.208 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-volume.log: 88.0% -- replaced with /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T02:39:52.208 INFO:teuthology.orchestra.run.smithi173.stderr: 92.3% -- replaced with /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T02:39:52.211 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-mon.b.log: 92.6% -- replaced with /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-mgr.x.log.gz 2024-03-26T02:39:52.213 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-osd.0.log: 95.3% -- replaced with /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T02:39:52.213 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph.log 2024-03-26T02:39:52.214 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph.cephadm.log: 89.0% -- replaced with /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T02:39:52.216 INFO:teuthology.orchestra.run.smithi173.stderr: 91.0% -- replaced with /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T02:39:52.223 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-26T02:39:52.232 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T02:39:52.234 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-osd.2.log: 93.4% -- replaced with /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T02:39:52.234 INFO:teuthology.orchestra.run.smithi072.stderr:gzip -5 --verbose -- /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-mon.c.log 2024-03-26T02:39:52.249 INFO:teuthology.orchestra.run.smithi072.stderr:/var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-mon.c.log: 91.0% -- replaced with /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T02:39:52.262 INFO:teuthology.orchestra.run.smithi072.stderr: 94.1% -- replaced with /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-26T02:39:52.755 INFO:teuthology.orchestra.run.smithi072.stderr: 90.7% -- replaced with /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-mgr.y.log.gz 2024-03-26T02:39:53.488 INFO:teuthology.orchestra.run.smithi072.stderr: 92.7% -- replaced with /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-mon.c.log.gz 2024-03-26T02:39:53.810 INFO:teuthology.orchestra.run.smithi173.stderr: 92.9% -- replaced with /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-mon.b.log.gz 2024-03-26T02:39:54.711 INFO:teuthology.orchestra.run.smithi072.stderr: 91.8% -- replaced with /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-mon.a.log.gz 2024-03-26T02:40:00.583 INFO:teuthology.orchestra.run.smithi173.stderr: 93.9% -- replaced with /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-osd.5.log.gz 2024-03-26T02:40:00.750 INFO:teuthology.orchestra.run.smithi173.stderr: 93.9% -- replaced with /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-osd.6.log.gz 2024-03-26T02:40:00.767 INFO:teuthology.orchestra.run.smithi072.stderr: 93.9% -- replaced with /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-26T02:40:00.897 INFO:teuthology.orchestra.run.smithi173.stderr: 93.8% -- replaced with /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-osd.7.log.gz 2024-03-26T02:40:01.335 INFO:teuthology.orchestra.run.smithi072.stderr: 94.0% -- replaced with /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-26T02:40:01.520 INFO:teuthology.orchestra.run.smithi072.stderr: 94.0% -- replaced with /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-26T02:40:01.606 INFO:teuthology.orchestra.run.smithi173.stderr: 94.0% -- replaced with /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-osd.4.log.gz 2024-03-26T02:40:01.608 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-03-26T02:40:01.608 INFO:teuthology.orchestra.run.smithi173.stderr:real 0m9.425s 2024-03-26T02:40:01.608 INFO:teuthology.orchestra.run.smithi173.stderr:user 0m35.411s 2024-03-26T02:40:01.608 INFO:teuthology.orchestra.run.smithi173.stderr:sys 0m1.250s 2024-03-26T02:40:01.766 INFO:teuthology.orchestra.run.smithi072.stderr: 94.0% -- replaced with /var/log/ceph/c86fbdd4-eb13-11ee-95d0-87774f69a715/ceph-osd.3.log.gz 2024-03-26T02:40:01.767 INFO:teuthology.orchestra.run.smithi072.stderr: 2024-03-26T02:40:01.767 INFO:teuthology.orchestra.run.smithi072.stderr:real 0m9.587s 2024-03-26T02:40:01.768 INFO:teuthology.orchestra.run.smithi072.stderr:user 0m39.528s 2024-03-26T02:40:01.768 INFO:teuthology.orchestra.run.smithi072.stderr:sys 0m1.383s 2024-03-26T02:40:01.768 INFO:tasks.cephadm:Archiving logs... 2024-03-26T02:40:01.768 DEBUG:teuthology.misc:Transferring archived files from smithi072:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621931/remote/smithi072/log 2024-03-26T02:40:01.769 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T02:40:07.373 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621931/remote/smithi173/log 2024-03-26T02:40:07.374 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T02:40:12.459 INFO:tasks.cephadm:Removing cluster... 2024-03-26T02:40:12.460 DEBUG:teuthology.orchestra.run.smithi072:> sudo cephadm rm-cluster --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 --force 2024-03-26T02:40:12.652 INFO:teuthology.orchestra.run.smithi072.stdout:Deleting cluster with fsid: c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T02:40:12.948 DEBUG:teuthology.orchestra.run.smithi173:> sudo cephadm rm-cluster --fsid c86fbdd4-eb13-11ee-95d0-87774f69a715 --force 2024-03-26T02:40:13.142 INFO:teuthology.orchestra.run.smithi173.stdout:Deleting cluster with fsid: c86fbdd4-eb13-11ee-95d0-87774f69a715 2024-03-26T02:40:13.445 INFO:tasks.cephadm:Teardown complete 2024-03-26T02:40:13.446 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-26T02:40:13.456 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-26T02:40:13.456 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-26T02:40:13.459 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-26T02:40:13.529 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-26T02:40:13.538 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-26T02:40:13.538 DEBUG:teuthology.orchestra.run.smithi072:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T02:40:13.541 DEBUG:teuthology.orchestra.run.smithi173:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T02:40:13.553 INFO:teuthology.orchestra.run.smithi072.stderr:bash: line 1: ntpq: command not found 2024-03-26T02:40:13.554 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-03-26T02:40:13.557 INFO:teuthology.orchestra.run.smithi072.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T02:40:13.557 INFO:teuthology.orchestra.run.smithi072.stdout:=============================================================================== 2024-03-26T02:40:13.557 INFO:teuthology.orchestra.run.smithi072.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 58 -361us[ -361us] +/- 66ms 2024-03-26T02:40:13.557 INFO:teuthology.orchestra.run.smithi072.stdout:^* hv02.front.sepia.ceph.com 3 6 377 61 -487us[ -488us] +/- 54ms 2024-03-26T02:40:13.557 INFO:teuthology.orchestra.run.smithi072.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 59 +869us[ +869us] +/- 68ms 2024-03-26T02:40:13.557 INFO:teuthology.orchestra.run.smithi072.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:40:13.558 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T02:40:13.558 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-03-26T02:40:13.558 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 58 -365us[ -383us] +/- 66ms 2024-03-26T02:40:13.558 INFO:teuthology.orchestra.run.smithi173.stdout:^* hv02.front.sepia.ceph.com 3 6 377 58 -481us[ -499us] +/- 54ms 2024-03-26T02:40:13.559 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 63 +883us[ +865us] +/- 68ms 2024-03-26T02:40:13.559 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T02:40:13.559 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-26T02:40:13.568 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-26T02:40:13.568 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-26T02:40:13.577 DEBUG:teuthology.orchestra.run.smithi072:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T02:40:13.680 DEBUG:teuthology.orchestra.run.smithi173:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T02:40:13.755 DEBUG:teuthology.orchestra.run.smithi072:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T02:40:13.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:40:13.786 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T02:40:13.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:40:13.816 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-26T02:40:13.828 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-26T02:40:13.862 INFO:teuthology.task.internal:Duration was 3184.756630 seconds 2024-03-26T02:40:13.863 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-26T02:40:13.872 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-26T02:40:13.872 DEBUG:teuthology.orchestra.run.smithi072:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T02:40:13.874 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T02:40:13.915 INFO:teuthology.orchestra.run.smithi072.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T02:40:13.917 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T02:40:14.260 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-26T02:40:14.260 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi072.front.sepia.ceph.com 2024-03-26T02:40:14.261 DEBUG:teuthology.orchestra.run.smithi072:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T02:40:14.320 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi173.front.sepia.ceph.com 2024-03-26T02:40:14.320 DEBUG:teuthology.orchestra.run.smithi173:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T02:40:14.341 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-26T02:40:14.341 DEBUG:teuthology.orchestra.run.smithi072:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T02:40:14.363 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T02:40:14.411 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-26T02:40:14.411 DEBUG:teuthology.orchestra.run.smithi072:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T02:40:14.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T02:40:14.895 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-26T02:40:14.904 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-26T02:40:14.904 DEBUG:teuthology.orchestra.run.smithi072:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T02:40:14.931 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T02:40:14.958 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-26T02:40:14.968 DEBUG:teuthology.orchestra.run.smithi072:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T02:40:14.974 DEBUG:teuthology.orchestra.run.smithi173:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T02:40:14.997 INFO:teuthology.orchestra.run.smithi072.stdout:kernel.core_pattern = core 2024-03-26T02:40:15.024 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = core 2024-03-26T02:40:15.044 DEBUG:teuthology.orchestra.run.smithi072:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T02:40:15.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:40:15.066 DEBUG:teuthology.orchestra.run.smithi173:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T02:40:15.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T02:40:15.097 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-26T02:40:15.108 INFO:teuthology.task.internal:Transferring archived files... 2024-03-26T02:40:15.108 DEBUG:teuthology.misc:Transferring archived files from smithi072:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621931/remote/smithi072 2024-03-26T02:40:15.109 DEBUG:teuthology.orchestra.run.smithi072:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T02:40:15.173 DEBUG:teuthology.misc:Transferring archived files from smithi173:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621931/remote/smithi173 2024-03-26T02:40:15.173 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T02:40:15.213 INFO:teuthology.task.internal:Removing archive directory... 2024-03-26T02:40:15.213 DEBUG:teuthology.orchestra.run.smithi072:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T02:40:15.215 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T02:40:15.261 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-26T02:40:15.274 INFO:teuthology.task.internal:Not uploading archives. 2024-03-26T02:40:15.274 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-26T02:40:15.284 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-26T02:40:15.284 DEBUG:teuthology.orchestra.run.smithi072:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T02:40:15.286 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T02:40:15.298 INFO:teuthology.orchestra.run.smithi072.stdout: 265594 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 02:40 /home/ubuntu/cephtest 2024-03-26T02:40:15.316 INFO:teuthology.orchestra.run.smithi173.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 02:40 /home/ubuntu/cephtest 2024-03-26T02:40:15.317 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-26T02:40:15.327 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-26T02:40:15.346 INFO:teuthology.nuke:Checking targets against current locks 2024-03-26T02:40:15.369 DEBUG:teuthology.nuke:shortname: smithi072 2024-03-26T02:40:15.369 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T02:40:15.388 DEBUG:teuthology.nuke:shortname: smithi173 2024-03-26T02:40:15.388 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T02:40:15.392 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi072.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621931', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 01:38:27.513298', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ5gnemEUL2Ea2CURc2ZLotym3ZebY8KEH1+QZ84LeAnn4Sarfk9WPTqPEhmxdxR9RWS89EgCRi48t56sFPdlb4='} 2024-03-26T02:40:15.408 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621931', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 01:38:27.511347', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJR4IJEppupGoPqwaxFcNA3mN75efbtlVIdC+QPBtGM33IkD91QvLX0vXCubmrVNl0+RXcjRCikGpXKl85e5xuM='} 2024-03-26T02:40:15.433 INFO:teuthology.orchestra.console.smithi072:Power off 2024-03-26T02:40:15.433 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T02:40:15.458 DEBUG:teuthology.orchestra.console.smithi072:power off output: Chassis Power Control: Down/Off 2024-03-26T02:40:15.459 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:40:15.472 INFO:teuthology.orchestra.console.smithi173:Power off 2024-03-26T02:40:15.472 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T02:40:15.483 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2024-03-26T02:40:15.492 DEBUG:teuthology.orchestra.console.smithi173:power off output: Chassis Power Control: Down/Off 2024-03-26T02:40:15.493 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:40:15.517 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-03-26T02:40:19.485 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:40:19.518 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:40:19.595 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is on 2024-03-26T02:40:19.629 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is on 2024-03-26T02:40:23.600 DEBUG:teuthology.orchestra.console.smithi072:pexpect command: ipmitool -H smithi072.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:40:23.630 DEBUG:teuthology.orchestra.console.smithi173:pexpect command: ipmitool -H smithi173.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T02:40:23.712 DEBUG:teuthology.orchestra.console.smithi072:check power output: Chassis Power is off 2024-03-26T02:40:23.741 DEBUG:teuthology.orchestra.console.smithi173:check power output: Chassis Power is off 2024-03-26T02:40:23.813 INFO:teuthology.orchestra.console.smithi072:Power off completed 2024-03-26T02:40:23.842 INFO:teuthology.orchestra.console.smithi173:Power off completed 2024-03-26T02:40:23.946 INFO:teuthology.run:Summary data: description: orch/cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/connectivity msgr/async-v2only start tasks/rados_api_tests} duration: 3184.756629705429 failure_reason: '"2024-03-26T02:10:00.000151+0000 mon.a (mon.0) 2538 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 4 pool(s) do not have an application enabled" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-03-26T02:40:23.947 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T02:40:24.032 INFO:teuthology.run:FAIL